2023-04-27 03:53:32,549 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19 2023-04-27 03:53:32,564 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-27 03:53:32,580 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-27 03:53:32,581 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/cluster_10dc1395-631d-3033-a522-9ded50d65883, deleteOnExit=true 2023-04-27 03:53:32,581 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-27 03:53:32,581 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/test.cache.data in system properties and HBase conf 2023-04-27 03:53:32,582 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/hadoop.tmp.dir in system properties and HBase conf 2023-04-27 03:53:32,582 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/hadoop.log.dir in system properties and HBase conf 2023-04-27 03:53:32,583 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-27 03:53:32,584 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-27 03:53:32,584 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-27 03:53:32,705 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-27 03:53:33,092 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-27 03:53:33,096 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-27 03:53:33,097 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-27 03:53:33,097 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-27 03:53:33,097 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-27 03:53:33,098 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-27 03:53:33,098 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-27 03:53:33,098 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-27 03:53:33,098 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-27 03:53:33,099 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-27 03:53:33,099 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/nfs.dump.dir in system properties and HBase conf 2023-04-27 03:53:33,100 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/java.io.tmpdir in system properties and HBase conf 2023-04-27 03:53:33,100 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-27 03:53:33,101 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-27 03:53:33,101 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-27 03:53:33,688 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-27 03:53:33,692 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-27 03:53:33,992 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-27 03:53:34,185 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-27 03:53:34,205 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-27 03:53:34,247 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-27 03:53:34,280 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/java.io.tmpdir/Jetty_localhost_40039_hdfs____.l4phoc/webapp 2023-04-27 03:53:34,425 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40039 2023-04-27 03:53:34,483 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-27 03:53:34,484 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-27 03:53:34,914 WARN [Listener at localhost/33677] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-27 03:53:35,054 WARN [Listener at localhost/33677] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-27 03:53:35,075 WARN [Listener at localhost/33677] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-27 03:53:35,083 INFO [Listener at localhost/33677] log.Slf4jLog(67): jetty-6.1.26 2023-04-27 03:53:35,088 INFO [Listener at localhost/33677] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/java.io.tmpdir/Jetty_localhost_37029_datanode____dbnr3u/webapp 2023-04-27 03:53:35,196 INFO [Listener at localhost/33677] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37029 2023-04-27 03:53:35,569 WARN [Listener at localhost/33933] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-27 03:53:36,150 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x169207dc09e67ee9: Processing first storage report for DS-d022dc65-669e-4f21-bf35-4f576e77d6aa from datanode 689edc05-2e05-4ce0-af8a-6f976cd8f04d 2023-04-27 03:53:36,152 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x169207dc09e67ee9: from storage DS-d022dc65-669e-4f21-bf35-4f576e77d6aa node DatanodeRegistration(127.0.0.1:46059, datanodeUuid=689edc05-2e05-4ce0-af8a-6f976cd8f04d, infoPort=44327, infoSecurePort=0, ipcPort=33933, storageInfo=lv=-57;cid=testClusterID;nsid=1300959310;c=1682567613764), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-27 03:53:36,152 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x169207dc09e67ee9: Processing first storage report for DS-4f498d05-8eaf-46a6-ba75-31839db8c576 from datanode 689edc05-2e05-4ce0-af8a-6f976cd8f04d 2023-04-27 03:53:36,152 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x169207dc09e67ee9: from storage DS-4f498d05-8eaf-46a6-ba75-31839db8c576 node DatanodeRegistration(127.0.0.1:46059, datanodeUuid=689edc05-2e05-4ce0-af8a-6f976cd8f04d, infoPort=44327, infoSecurePort=0, ipcPort=33933, storageInfo=lv=-57;cid=testClusterID;nsid=1300959310;c=1682567613764), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-27 03:53:36,165 DEBUG [Listener at localhost/33933] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19 2023-04-27 03:53:36,251 INFO [Listener at localhost/33933] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/cluster_10dc1395-631d-3033-a522-9ded50d65883/zookeeper_0, clientPort=55020, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/cluster_10dc1395-631d-3033-a522-9ded50d65883/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/cluster_10dc1395-631d-3033-a522-9ded50d65883/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-27 03:53:36,264 INFO [Listener at localhost/33933] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=55020 2023-04-27 03:53:36,276 INFO [Listener at localhost/33933] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-27 03:53:36,278 INFO [Listener at localhost/33933] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-27 03:53:36,935 INFO [Listener at localhost/33933] util.FSUtils(471): Created version file at hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421 with version=8 2023-04-27 03:53:36,936 INFO [Listener at localhost/33933] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/hbase-staging 2023-04-27 03:53:37,247 INFO [Listener at localhost/33933] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-27 03:53:37,723 INFO [Listener at localhost/33933] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-27 03:53:37,755 INFO [Listener at localhost/33933] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-27 03:53:37,755 INFO [Listener at localhost/33933] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-27 03:53:37,755 INFO [Listener at localhost/33933] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-27 03:53:37,756 INFO [Listener at localhost/33933] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-27 03:53:37,756 INFO [Listener at localhost/33933] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-27 03:53:37,914 INFO [Listener at localhost/33933] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-27 03:53:38,008 DEBUG [Listener at localhost/33933] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-27 03:53:38,101 INFO [Listener at localhost/33933] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39131 2023-04-27 03:53:38,112 INFO [Listener at localhost/33933] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-27 03:53:38,114 INFO [Listener at localhost/33933] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-27 03:53:38,137 INFO [Listener at localhost/33933] zookeeper.RecoverableZooKeeper(93): Process identifier=master:39131 connecting to ZooKeeper ensemble=127.0.0.1:55020 2023-04-27 03:53:38,184 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:391310x0, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-27 03:53:38,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:39131-0x101c3c040af0000 connected 2023-04-27 03:53:38,218 DEBUG [Listener at localhost/33933] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-27 03:53:38,219 DEBUG [Listener at localhost/33933] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-27 03:53:38,222 DEBUG [Listener at localhost/33933] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-27 03:53:38,230 DEBUG [Listener at localhost/33933] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39131 2023-04-27 03:53:38,231 DEBUG [Listener at localhost/33933] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39131 2023-04-27 03:53:38,231 DEBUG [Listener at localhost/33933] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39131 2023-04-27 03:53:38,231 DEBUG [Listener at localhost/33933] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39131 2023-04-27 03:53:38,232 DEBUG [Listener at localhost/33933] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39131 2023-04-27 03:53:38,238 INFO [Listener at localhost/33933] master.HMaster(444): hbase.rootdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421, hbase.cluster.distributed=false 2023-04-27 03:53:38,306 INFO [Listener at localhost/33933] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-27 03:53:38,306 INFO [Listener at localhost/33933] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-27 03:53:38,306 INFO [Listener at localhost/33933] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-27 03:53:38,306 INFO [Listener at localhost/33933] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-27 03:53:38,307 INFO [Listener at localhost/33933] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-27 03:53:38,307 INFO [Listener at localhost/33933] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-27 03:53:38,311 INFO [Listener at localhost/33933] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-27 03:53:38,314 INFO [Listener at localhost/33933] ipc.NettyRpcServer(120): Bind to /172.31.14.131:40885 2023-04-27 03:53:38,316 INFO [Listener at localhost/33933] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-27 03:53:38,324 DEBUG [Listener at localhost/33933] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-27 03:53:38,325 INFO [Listener at localhost/33933] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-27 03:53:38,328 INFO [Listener at localhost/33933] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-27 03:53:38,329 INFO [Listener at localhost/33933] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:40885 connecting to ZooKeeper ensemble=127.0.0.1:55020 2023-04-27 03:53:38,333 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:408850x0, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-27 03:53:38,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:40885-0x101c3c040af0001 connected 2023-04-27 03:53:38,335 DEBUG [Listener at localhost/33933] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-27 03:53:38,336 DEBUG [Listener at localhost/33933] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-27 03:53:38,337 DEBUG [Listener at localhost/33933] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-27 03:53:38,340 DEBUG [Listener at localhost/33933] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40885 2023-04-27 03:53:38,340 DEBUG [Listener at localhost/33933] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40885 2023-04-27 03:53:38,341 DEBUG [Listener at localhost/33933] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40885 2023-04-27 03:53:38,342 DEBUG [Listener at localhost/33933] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40885 2023-04-27 03:53:38,343 DEBUG [Listener at localhost/33933] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40885 2023-04-27 03:53:38,348 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,39131,1682567617081 2023-04-27 03:53:38,358 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-27 03:53:38,360 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,39131,1682567617081 2023-04-27 03:53:38,379 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-27 03:53:38,379 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-27 03:53:38,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-27 03:53:38,380 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-27 03:53:38,382 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,39131,1682567617081 from backup master directory 2023-04-27 03:53:38,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-27 03:53:38,385 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,39131,1682567617081 2023-04-27 03:53:38,386 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-27 03:53:38,386 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-27 03:53:38,387 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,39131,1682567617081 2023-04-27 03:53:38,390 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-27 03:53:38,391 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-27 03:53:38,485 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/hbase.id with ID: 32b80d5a-13aa-4444-a59e-606e93931ad4 2023-04-27 03:53:38,532 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-27 03:53:38,550 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-27 03:53:38,591 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x53fd259f to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:53:38,629 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e5f79e6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:53:38,658 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-27 03:53:38,661 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-27 03:53:38,682 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-27 03:53:38,682 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-27 03:53:38,684 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-27 03:53:38,689 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-27 03:53:38,690 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-27 03:53:38,729 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/MasterData/data/master/store-tmp 2023-04-27 03:53:38,758 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 03:53:38,758 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-27 03:53:38,758 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-27 03:53:38,758 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-27 03:53:38,758 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-27 03:53:38,758 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-27 03:53:38,758 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-27 03:53:38,759 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-27 03:53:38,760 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/MasterData/WALs/jenkins-hbase4.apache.org,39131,1682567617081 2023-04-27 03:53:38,784 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39131%2C1682567617081, suffix=, logDir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/MasterData/WALs/jenkins-hbase4.apache.org,39131,1682567617081, archiveDir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/MasterData/oldWALs, maxLogs=10 2023-04-27 03:53:38,843 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46059,DS-d022dc65-669e-4f21-bf35-4f576e77d6aa,DISK] 2023-04-27 03:53:38,852 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 03:53:38,921 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/MasterData/WALs/jenkins-hbase4.apache.org,39131,1682567617081/jenkins-hbase4.apache.org%2C39131%2C1682567617081.1682567618794 2023-04-27 03:53:38,922 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46059,DS-d022dc65-669e-4f21-bf35-4f576e77d6aa,DISK]] 2023-04-27 03:53:38,922 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-27 03:53:38,923 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 03:53:38,926 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-27 03:53:38,928 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-27 03:53:38,989 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-27 03:53:38,995 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-27 03:53:39,019 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-27 03:53:39,032 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:53:39,039 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-27 03:53:39,045 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-27 03:53:39,063 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-27 03:53:39,068 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-27 03:53:39,069 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68640730, jitterRate=0.022826582193374634}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-27 03:53:39,069 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-27 03:53:39,070 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-27 03:53:39,091 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-27 03:53:39,092 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-27 03:53:39,095 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-27 03:53:39,097 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-27 03:53:39,138 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 40 msec 2023-04-27 03:53:39,139 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-27 03:53:39,167 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-27 03:53:39,173 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-27 03:53:39,203 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-27 03:53:39,208 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-27 03:53:39,211 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-27 03:53:39,217 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-27 03:53:39,222 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-27 03:53:39,225 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-27 03:53:39,227 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-27 03:53:39,231 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-27 03:53:39,244 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-27 03:53:39,250 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-27 03:53:39,250 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-27 03:53:39,251 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-27 03:53:39,252 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,39131,1682567617081, sessionid=0x101c3c040af0000, setting cluster-up flag (Was=false) 2023-04-27 03:53:39,269 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-27 03:53:39,275 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-27 03:53:39,277 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,39131,1682567617081 2023-04-27 03:53:39,284 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-27 03:53:39,291 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-27 03:53:39,295 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,39131,1682567617081 2023-04-27 03:53:39,298 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.hbase-snapshot/.tmp 2023-04-27 03:53:39,348 INFO [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(952): ClusterId : 32b80d5a-13aa-4444-a59e-606e93931ad4 2023-04-27 03:53:39,354 DEBUG [RS:0;jenkins-hbase4:40885] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-27 03:53:39,361 DEBUG [RS:0;jenkins-hbase4:40885] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-27 03:53:39,361 DEBUG [RS:0;jenkins-hbase4:40885] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-27 03:53:39,364 DEBUG [RS:0;jenkins-hbase4:40885] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-27 03:53:39,365 DEBUG [RS:0;jenkins-hbase4:40885] zookeeper.ReadOnlyZKClient(139): Connect 0x006e5da7 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:53:39,376 DEBUG [RS:0;jenkins-hbase4:40885] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ea9a3da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:53:39,376 DEBUG [RS:0;jenkins-hbase4:40885] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a9636d7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-27 03:53:39,402 DEBUG [RS:0;jenkins-hbase4:40885] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:40885 2023-04-27 03:53:39,406 INFO [RS:0;jenkins-hbase4:40885] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-27 03:53:39,406 INFO [RS:0;jenkins-hbase4:40885] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-27 03:53:39,406 DEBUG [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(1023): About to register with Master. 2023-04-27 03:53:39,409 INFO [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,39131,1682567617081 with isa=jenkins-hbase4.apache.org/172.31.14.131:40885, startcode=1682567618305 2023-04-27 03:53:39,431 DEBUG [RS:0;jenkins-hbase4:40885] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-27 03:53:39,448 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-27 03:53:39,462 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-27 03:53:39,462 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-27 03:53:39,462 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-27 03:53:39,462 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-27 03:53:39,463 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-27 03:53:39,463 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-27 03:53:39,463 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-27 03:53:39,463 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-27 03:53:39,479 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682567649479 2023-04-27 03:53:39,482 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-27 03:53:39,486 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-27 03:53:39,487 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-27 03:53:39,493 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-27 03:53:39,493 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-27 03:53:39,498 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57795, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-27 03:53:39,504 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-27 03:53:39,504 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-27 03:53:39,505 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-27 03:53:39,505 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-27 03:53:39,506 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-27 03:53:39,507 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-27 03:53:39,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:39,509 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-27 03:53:39,509 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-27 03:53:39,513 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-27 03:53:39,513 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-27 03:53:39,516 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682567619515,5,FailOnTimeoutGroup] 2023-04-27 03:53:39,516 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682567619516,5,FailOnTimeoutGroup] 2023-04-27 03:53:39,516 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-27 03:53:39,516 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-27 03:53:39,519 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-27 03:53:39,519 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-27 03:53:39,531 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-27 03:53:39,533 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-27 03:53:39,533 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421 2023-04-27 03:53:39,546 DEBUG [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(2829): Master is not running yet 2023-04-27 03:53:39,546 WARN [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-04-27 03:53:39,647 INFO [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,39131,1682567617081 with isa=jenkins-hbase4.apache.org/172.31.14.131:40885, startcode=1682567618305 2023-04-27 03:53:39,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:39,659 DEBUG [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421 2023-04-27 03:53:39,659 DEBUG [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:33677 2023-04-27 03:53:39,659 DEBUG [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-27 03:53:39,664 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-27 03:53:39,665 DEBUG [RS:0;jenkins-hbase4:40885] zookeeper.ZKUtil(162): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:39,666 WARN [RS:0;jenkins-hbase4:40885] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-27 03:53:39,666 INFO [RS:0;jenkins-hbase4:40885] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-27 03:53:39,667 DEBUG [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(1947): logDir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/WALs/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:39,667 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,40885,1682567618305] 2023-04-27 03:53:39,677 DEBUG [RS:0;jenkins-hbase4:40885] zookeeper.ZKUtil(162): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:39,689 DEBUG [RS:0;jenkins-hbase4:40885] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-27 03:53:39,698 INFO [RS:0;jenkins-hbase4:40885] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-27 03:53:39,717 INFO [RS:0;jenkins-hbase4:40885] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-27 03:53:39,721 INFO [RS:0;jenkins-hbase4:40885] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-27 03:53:39,721 INFO [RS:0;jenkins-hbase4:40885] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-27 03:53:39,723 INFO [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-27 03:53:39,730 INFO [RS:0;jenkins-hbase4:40885] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-27 03:53:39,730 DEBUG [RS:0;jenkins-hbase4:40885] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-27 03:53:39,730 DEBUG [RS:0;jenkins-hbase4:40885] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-27 03:53:39,730 DEBUG [RS:0;jenkins-hbase4:40885] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-27 03:53:39,730 DEBUG [RS:0;jenkins-hbase4:40885] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-27 03:53:39,731 DEBUG [RS:0;jenkins-hbase4:40885] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-27 03:53:39,731 DEBUG [RS:0;jenkins-hbase4:40885] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-27 03:53:39,731 DEBUG [RS:0;jenkins-hbase4:40885] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-27 03:53:39,731 DEBUG [RS:0;jenkins-hbase4:40885] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-27 03:53:39,731 DEBUG [RS:0;jenkins-hbase4:40885] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-27 03:53:39,731 DEBUG [RS:0;jenkins-hbase4:40885] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-27 03:53:39,732 INFO [RS:0;jenkins-hbase4:40885] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-27 03:53:39,732 INFO [RS:0;jenkins-hbase4:40885] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-27 03:53:39,732 INFO [RS:0;jenkins-hbase4:40885] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-27 03:53:39,753 INFO [RS:0;jenkins-hbase4:40885] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-27 03:53:39,756 INFO [RS:0;jenkins-hbase4:40885] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40885,1682567618305-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-27 03:53:39,773 INFO [RS:0;jenkins-hbase4:40885] regionserver.Replication(203): jenkins-hbase4.apache.org,40885,1682567618305 started 2023-04-27 03:53:39,774 INFO [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,40885,1682567618305, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:40885, sessionid=0x101c3c040af0001 2023-04-27 03:53:39,774 DEBUG [RS:0;jenkins-hbase4:40885] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-27 03:53:39,774 DEBUG [RS:0;jenkins-hbase4:40885] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:39,774 DEBUG [RS:0;jenkins-hbase4:40885] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,40885,1682567618305' 2023-04-27 03:53:39,774 DEBUG [RS:0;jenkins-hbase4:40885] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:39,775 DEBUG [RS:0;jenkins-hbase4:40885] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:39,775 DEBUG [RS:0;jenkins-hbase4:40885] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-27 03:53:39,776 DEBUG [RS:0;jenkins-hbase4:40885] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-27 03:53:39,776 DEBUG [RS:0;jenkins-hbase4:40885] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:39,776 DEBUG [RS:0;jenkins-hbase4:40885] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,40885,1682567618305' 2023-04-27 03:53:39,776 DEBUG [RS:0;jenkins-hbase4:40885] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-27 03:53:39,776 DEBUG [RS:0;jenkins-hbase4:40885] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-27 03:53:39,777 DEBUG [RS:0;jenkins-hbase4:40885] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-27 03:53:39,777 INFO [RS:0;jenkins-hbase4:40885] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-27 03:53:39,777 INFO [RS:0;jenkins-hbase4:40885] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-27 03:53:39,890 INFO [RS:0;jenkins-hbase4:40885] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40885%2C1682567618305, suffix=, logDir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/WALs/jenkins-hbase4.apache.org,40885,1682567618305, archiveDir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/oldWALs, maxLogs=32 2023-04-27 03:53:39,911 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46059,DS-d022dc65-669e-4f21-bf35-4f576e77d6aa,DISK] 2023-04-27 03:53:39,915 INFO [RS:0;jenkins-hbase4:40885] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/WALs/jenkins-hbase4.apache.org,40885,1682567618305/jenkins-hbase4.apache.org%2C40885%2C1682567618305.1682567619893 2023-04-27 03:53:39,916 DEBUG [RS:0;jenkins-hbase4:40885] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46059,DS-d022dc65-669e-4f21-bf35-4f576e77d6aa,DISK]] 2023-04-27 03:53:39,950 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 03:53:39,953 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-27 03:53:39,957 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740/info 2023-04-27 03:53:39,958 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-27 03:53:39,959 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:53:39,960 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-27 03:53:39,963 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740/rep_barrier 2023-04-27 03:53:39,964 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-27 03:53:39,964 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:53:39,965 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-27 03:53:39,967 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740/table 2023-04-27 03:53:39,968 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-27 03:53:39,969 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:53:39,971 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740 2023-04-27 03:53:39,972 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740 2023-04-27 03:53:39,977 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-27 03:53:39,979 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-27 03:53:39,983 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-27 03:53:39,984 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72315524, jitterRate=0.07758527994155884}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-27 03:53:39,984 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-27 03:53:39,985 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-27 03:53:39,985 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-27 03:53:39,985 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-27 03:53:39,985 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-27 03:53:39,985 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-27 03:53:39,986 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-27 03:53:39,987 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-27 03:53:39,995 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-27 03:53:39,995 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-27 03:53:40,004 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-27 03:53:40,018 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-27 03:53:40,021 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-27 03:53:40,173 DEBUG [jenkins-hbase4:39131] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-27 03:53:40,177 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,40885,1682567618305, state=OPENING 2023-04-27 03:53:40,184 DEBUG [PEWorker-4] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-27 03:53:40,185 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-27 03:53:40,186 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-27 03:53:40,188 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,40885,1682567618305}] 2023-04-27 03:53:40,370 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:40,373 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-27 03:53:40,376 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54756, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-27 03:53:40,387 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-27 03:53:40,388 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-27 03:53:40,391 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40885%2C1682567618305.meta, suffix=.meta, logDir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/WALs/jenkins-hbase4.apache.org,40885,1682567618305, archiveDir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/oldWALs, maxLogs=32 2023-04-27 03:53:40,408 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46059,DS-d022dc65-669e-4f21-bf35-4f576e77d6aa,DISK] 2023-04-27 03:53:40,413 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/WALs/jenkins-hbase4.apache.org,40885,1682567618305/jenkins-hbase4.apache.org%2C40885%2C1682567618305.meta.1682567620392.meta 2023-04-27 03:53:40,414 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46059,DS-d022dc65-669e-4f21-bf35-4f576e77d6aa,DISK]] 2023-04-27 03:53:40,414 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-27 03:53:40,416 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-27 03:53:40,433 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-27 03:53:40,439 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-27 03:53:40,446 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-27 03:53:40,446 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 03:53:40,446 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-27 03:53:40,446 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-27 03:53:40,450 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-27 03:53:40,453 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740/info 2023-04-27 03:53:40,453 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740/info 2023-04-27 03:53:40,453 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-27 03:53:40,455 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:53:40,455 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-27 03:53:40,456 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740/rep_barrier 2023-04-27 03:53:40,456 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740/rep_barrier 2023-04-27 03:53:40,457 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-27 03:53:40,458 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:53:40,458 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-27 03:53:40,460 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740/table 2023-04-27 03:53:40,460 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740/table 2023-04-27 03:53:40,461 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-27 03:53:40,462 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:53:40,463 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740 2023-04-27 03:53:40,466 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740 2023-04-27 03:53:40,471 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-27 03:53:40,473 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-27 03:53:40,475 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60267266, jitterRate=-0.10194775462150574}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-27 03:53:40,475 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-27 03:53:40,485 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1682567620363 2023-04-27 03:53:40,504 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-27 03:53:40,505 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,40885,1682567618305, state=OPEN 2023-04-27 03:53:40,505 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-27 03:53:40,507 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-27 03:53:40,508 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-27 03:53:40,512 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-27 03:53:40,512 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,40885,1682567618305 in 319 msec 2023-04-27 03:53:40,518 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-27 03:53:40,518 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 510 msec 2023-04-27 03:53:40,524 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.1570 sec 2023-04-27 03:53:40,524 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1682567620524, completionTime=-1 2023-04-27 03:53:40,525 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-27 03:53:40,525 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-27 03:53:40,603 DEBUG [hconnection-0x6dfca63e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:53:40,607 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54764, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:53:40,628 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-27 03:53:40,628 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682567680628 2023-04-27 03:53:40,628 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682567740628 2023-04-27 03:53:40,629 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 103 msec 2023-04-27 03:53:40,653 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39131,1682567617081-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-27 03:53:40,653 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39131,1682567617081-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-27 03:53:40,653 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39131,1682567617081-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-27 03:53:40,655 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:39131, period=300000, unit=MILLISECONDS is enabled. 2023-04-27 03:53:40,655 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-27 03:53:40,662 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-27 03:53:40,670 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-27 03:53:40,671 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-27 03:53:40,681 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-27 03:53:40,683 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-27 03:53:40,686 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-27 03:53:40,711 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/hbase/namespace/bc58fe2fa2408f679ad2d4ab14224359 2023-04-27 03:53:40,714 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/hbase/namespace/bc58fe2fa2408f679ad2d4ab14224359 empty. 2023-04-27 03:53:40,715 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/hbase/namespace/bc58fe2fa2408f679ad2d4ab14224359 2023-04-27 03:53:40,715 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-27 03:53:41,140 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-27 03:53:41,142 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => bc58fe2fa2408f679ad2d4ab14224359, NAME => 'hbase:namespace,,1682567620670.bc58fe2fa2408f679ad2d4ab14224359.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp 2023-04-27 03:53:41,555 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682567620670.bc58fe2fa2408f679ad2d4ab14224359.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 03:53:41,556 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing bc58fe2fa2408f679ad2d4ab14224359, disabling compactions & flushes 2023-04-27 03:53:41,556 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682567620670.bc58fe2fa2408f679ad2d4ab14224359. 2023-04-27 03:53:41,556 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682567620670.bc58fe2fa2408f679ad2d4ab14224359. 2023-04-27 03:53:41,556 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682567620670.bc58fe2fa2408f679ad2d4ab14224359. after waiting 0 ms 2023-04-27 03:53:41,556 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682567620670.bc58fe2fa2408f679ad2d4ab14224359. 2023-04-27 03:53:41,556 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682567620670.bc58fe2fa2408f679ad2d4ab14224359. 2023-04-27 03:53:41,556 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for bc58fe2fa2408f679ad2d4ab14224359: 2023-04-27 03:53:41,561 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-27 03:53:41,576 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1682567620670.bc58fe2fa2408f679ad2d4ab14224359.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682567621564"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682567621564"}]},"ts":"1682567621564"} 2023-04-27 03:53:41,605 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-27 03:53:41,608 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-27 03:53:41,613 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567621608"}]},"ts":"1682567621608"} 2023-04-27 03:53:41,619 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-27 03:53:41,633 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=bc58fe2fa2408f679ad2d4ab14224359, ASSIGN}] 2023-04-27 03:53:41,636 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=bc58fe2fa2408f679ad2d4ab14224359, ASSIGN 2023-04-27 03:53:41,638 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=bc58fe2fa2408f679ad2d4ab14224359, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40885,1682567618305; forceNewPlan=false, retain=false 2023-04-27 03:53:41,789 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=bc58fe2fa2408f679ad2d4ab14224359, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:41,789 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1682567620670.bc58fe2fa2408f679ad2d4ab14224359.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682567621789"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682567621789"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682567621789"}]},"ts":"1682567621789"} 2023-04-27 03:53:41,793 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure bc58fe2fa2408f679ad2d4ab14224359, server=jenkins-hbase4.apache.org,40885,1682567618305}] 2023-04-27 03:53:41,954 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1682567620670.bc58fe2fa2408f679ad2d4ab14224359. 2023-04-27 03:53:41,955 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => bc58fe2fa2408f679ad2d4ab14224359, NAME => 'hbase:namespace,,1682567620670.bc58fe2fa2408f679ad2d4ab14224359.', STARTKEY => '', ENDKEY => ''} 2023-04-27 03:53:41,956 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace bc58fe2fa2408f679ad2d4ab14224359 2023-04-27 03:53:41,956 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682567620670.bc58fe2fa2408f679ad2d4ab14224359.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 03:53:41,956 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for bc58fe2fa2408f679ad2d4ab14224359 2023-04-27 03:53:41,956 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for bc58fe2fa2408f679ad2d4ab14224359 2023-04-27 03:53:41,958 INFO [StoreOpener-bc58fe2fa2408f679ad2d4ab14224359-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region bc58fe2fa2408f679ad2d4ab14224359 2023-04-27 03:53:41,961 DEBUG [StoreOpener-bc58fe2fa2408f679ad2d4ab14224359-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/namespace/bc58fe2fa2408f679ad2d4ab14224359/info 2023-04-27 03:53:41,961 DEBUG [StoreOpener-bc58fe2fa2408f679ad2d4ab14224359-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/namespace/bc58fe2fa2408f679ad2d4ab14224359/info 2023-04-27 03:53:41,961 INFO [StoreOpener-bc58fe2fa2408f679ad2d4ab14224359-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bc58fe2fa2408f679ad2d4ab14224359 columnFamilyName info 2023-04-27 03:53:41,962 INFO [StoreOpener-bc58fe2fa2408f679ad2d4ab14224359-1] regionserver.HStore(310): Store=bc58fe2fa2408f679ad2d4ab14224359/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:53:41,964 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/namespace/bc58fe2fa2408f679ad2d4ab14224359 2023-04-27 03:53:41,964 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/namespace/bc58fe2fa2408f679ad2d4ab14224359 2023-04-27 03:53:41,969 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for bc58fe2fa2408f679ad2d4ab14224359 2023-04-27 03:53:41,972 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/namespace/bc58fe2fa2408f679ad2d4ab14224359/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-27 03:53:41,973 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened bc58fe2fa2408f679ad2d4ab14224359; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63484188, jitterRate=-0.05401188135147095}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-27 03:53:41,973 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for bc58fe2fa2408f679ad2d4ab14224359: 2023-04-27 03:53:41,975 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1682567620670.bc58fe2fa2408f679ad2d4ab14224359., pid=6, masterSystemTime=1682567621948 2023-04-27 03:53:41,979 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1682567620670.bc58fe2fa2408f679ad2d4ab14224359. 2023-04-27 03:53:41,979 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1682567620670.bc58fe2fa2408f679ad2d4ab14224359. 2023-04-27 03:53:41,981 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=bc58fe2fa2408f679ad2d4ab14224359, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:41,981 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1682567620670.bc58fe2fa2408f679ad2d4ab14224359.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682567621980"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682567621980"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682567621980"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682567621980"}]},"ts":"1682567621980"} 2023-04-27 03:53:41,990 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-27 03:53:41,990 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure bc58fe2fa2408f679ad2d4ab14224359, server=jenkins-hbase4.apache.org,40885,1682567618305 in 193 msec 2023-04-27 03:53:41,993 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-27 03:53:41,994 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=bc58fe2fa2408f679ad2d4ab14224359, ASSIGN in 357 msec 2023-04-27 03:53:41,995 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-27 03:53:41,995 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567621995"}]},"ts":"1682567621995"} 2023-04-27 03:53:41,998 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-27 03:53:42,003 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-27 03:53:42,007 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3310 sec 2023-04-27 03:53:42,085 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-27 03:53:42,086 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-27 03:53:42,087 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-27 03:53:42,143 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-27 03:53:42,160 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-27 03:53:42,169 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 51 msec 2023-04-27 03:53:42,177 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-27 03:53:42,190 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-27 03:53:42,195 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 18 msec 2023-04-27 03:53:42,209 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-27 03:53:42,212 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-27 03:53:42,213 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.825sec 2023-04-27 03:53:42,214 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-27 03:53:42,216 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-27 03:53:42,216 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-27 03:53:42,217 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39131,1682567617081-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-27 03:53:42,218 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39131,1682567617081-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-27 03:53:42,229 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-27 03:53:42,257 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x3ddd91d3 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:53:42,262 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51f71277, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:53:42,276 DEBUG [hconnection-0x286751c0-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:53:42,291 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54774, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:53:42,302 INFO [Listener at localhost/33933] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,39131,1682567617081 2023-04-27 03:53:42,326 INFO [Listener at localhost/33933] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=207, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=299, ProcessCount=170, AvailableMemoryMB=12840 2023-04-27 03:53:42,345 DEBUG [Listener at localhost/33933] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-27 03:53:42,349 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52096, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-27 03:53:42,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-27 03:53:42,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-27 03:53:42,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-27 03:53:42,371 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-27 03:53:42,375 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-27 03:53:42,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-27 03:53:42,379 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:42,380 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35 empty. 2023-04-27 03:53:42,382 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:42,382 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-27 03:53:42,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-27 03:53:42,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-27 03:53:42,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-27 03:53:42,805 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-27 03:53:42,808 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2b04f11bbacd80b8750d75eb2d0fbb35, NAME => 'TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp 2023-04-27 03:53:43,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-27 03:53:43,225 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 03:53:43,226 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 2b04f11bbacd80b8750d75eb2d0fbb35, disabling compactions & flushes 2023-04-27 03:53:43,226 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:43,226 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:43,226 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. after waiting 0 ms 2023-04-27 03:53:43,226 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:43,226 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:43,226 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:43,231 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-27 03:53:43,233 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567623232"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682567623232"}]},"ts":"1682567623232"} 2023-04-27 03:53:43,236 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-27 03:53:43,237 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-27 03:53:43,237 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567623237"}]},"ts":"1682567623237"} 2023-04-27 03:53:43,240 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-27 03:53:43,244 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2b04f11bbacd80b8750d75eb2d0fbb35, ASSIGN}] 2023-04-27 03:53:43,246 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2b04f11bbacd80b8750d75eb2d0fbb35, ASSIGN 2023-04-27 03:53:43,247 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2b04f11bbacd80b8750d75eb2d0fbb35, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40885,1682567618305; forceNewPlan=false, retain=false 2023-04-27 03:53:43,399 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=2b04f11bbacd80b8750d75eb2d0fbb35, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:43,399 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567623399"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682567623399"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682567623399"}]},"ts":"1682567623399"} 2023-04-27 03:53:43,403 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305}] 2023-04-27 03:53:43,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-27 03:53:43,563 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:43,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2b04f11bbacd80b8750d75eb2d0fbb35, NAME => 'TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.', STARTKEY => '', ENDKEY => ''} 2023-04-27 03:53:43,564 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:43,564 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 03:53:43,564 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:43,564 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:43,566 INFO [StoreOpener-2b04f11bbacd80b8750d75eb2d0fbb35-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:43,569 DEBUG [StoreOpener-2b04f11bbacd80b8750d75eb2d0fbb35-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A 2023-04-27 03:53:43,569 DEBUG [StoreOpener-2b04f11bbacd80b8750d75eb2d0fbb35-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A 2023-04-27 03:53:43,570 INFO [StoreOpener-2b04f11bbacd80b8750d75eb2d0fbb35-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:53:43,571 INFO [StoreOpener-2b04f11bbacd80b8750d75eb2d0fbb35-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2b04f11bbacd80b8750d75eb2d0fbb35 columnFamilyName A 2023-04-27 03:53:43,572 INFO [StoreOpener-2b04f11bbacd80b8750d75eb2d0fbb35-1] regionserver.HStore(310): Store=2b04f11bbacd80b8750d75eb2d0fbb35/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:53:43,572 INFO [StoreOpener-2b04f11bbacd80b8750d75eb2d0fbb35-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:43,574 DEBUG [StoreOpener-2b04f11bbacd80b8750d75eb2d0fbb35-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B 2023-04-27 03:53:43,574 DEBUG [StoreOpener-2b04f11bbacd80b8750d75eb2d0fbb35-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B 2023-04-27 03:53:43,574 INFO [StoreOpener-2b04f11bbacd80b8750d75eb2d0fbb35-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:53:43,575 INFO [StoreOpener-2b04f11bbacd80b8750d75eb2d0fbb35-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2b04f11bbacd80b8750d75eb2d0fbb35 columnFamilyName B 2023-04-27 03:53:43,576 INFO [StoreOpener-2b04f11bbacd80b8750d75eb2d0fbb35-1] regionserver.HStore(310): Store=2b04f11bbacd80b8750d75eb2d0fbb35/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:53:43,576 INFO [StoreOpener-2b04f11bbacd80b8750d75eb2d0fbb35-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:43,579 DEBUG [StoreOpener-2b04f11bbacd80b8750d75eb2d0fbb35-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C 2023-04-27 03:53:43,579 DEBUG [StoreOpener-2b04f11bbacd80b8750d75eb2d0fbb35-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C 2023-04-27 03:53:43,579 INFO [StoreOpener-2b04f11bbacd80b8750d75eb2d0fbb35-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:53:43,579 INFO [StoreOpener-2b04f11bbacd80b8750d75eb2d0fbb35-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2b04f11bbacd80b8750d75eb2d0fbb35 columnFamilyName C 2023-04-27 03:53:43,580 INFO [StoreOpener-2b04f11bbacd80b8750d75eb2d0fbb35-1] regionserver.HStore(310): Store=2b04f11bbacd80b8750d75eb2d0fbb35/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:53:43,581 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:43,583 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:43,584 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:43,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-27 03:53:43,591 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:43,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-27 03:53:43,595 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2b04f11bbacd80b8750d75eb2d0fbb35; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73261126, jitterRate=0.09167584776878357}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-27 03:53:43,595 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:43,597 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., pid=11, masterSystemTime=1682567623556 2023-04-27 03:53:43,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:43,600 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:43,601 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=2b04f11bbacd80b8750d75eb2d0fbb35, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:43,601 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567623601"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682567623601"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682567623601"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682567623601"}]},"ts":"1682567623601"} 2023-04-27 03:53:43,608 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-27 03:53:43,608 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 in 201 msec 2023-04-27 03:53:43,611 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-27 03:53:43,612 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2b04f11bbacd80b8750d75eb2d0fbb35, ASSIGN in 364 msec 2023-04-27 03:53:43,613 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-27 03:53:43,613 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567623613"}]},"ts":"1682567623613"} 2023-04-27 03:53:43,616 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-27 03:53:43,620 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-27 03:53:43,623 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2540 sec 2023-04-27 03:53:44,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-27 03:53:44,505 INFO [Listener at localhost/33933] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-27 03:53:44,510 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x44ef43e3 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:53:44,517 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2678d495, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:53:44,520 DEBUG [Listener at localhost/33933] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:53:44,523 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50840, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:53:44,527 DEBUG [Listener at localhost/33933] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-27 03:53:44,530 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46952, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-27 03:53:44,539 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x61e72c30 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:53:44,544 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1b886c8f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:53:44,545 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x56c8e45e to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:53:44,549 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56223d42, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:53:44,549 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x4fea27db to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:53:44,553 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@77648431, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:53:44,554 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x6038a026 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:53:44,560 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2de4d1f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:53:44,560 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x63859c62 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:53:44,564 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9a575b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:53:44,564 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x49a45b6d to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:53:44,568 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1818ad0a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:53:44,568 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x6fef7a21 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:53:44,603 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fb2b82, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:53:44,604 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x3cd3abbd to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:53:44,610 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6129209e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:53:44,611 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x6aad2e18 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:53:44,615 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b02eeb3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:53:44,621 DEBUG [hconnection-0x2b90131a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:53:44,622 DEBUG [hconnection-0x4896b578-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:53:44,625 DEBUG [hconnection-0x139a00b7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:53:44,626 DEBUG [hconnection-0x2a12d69-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:53:44,626 DEBUG [hconnection-0x1330534b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:53:44,628 DEBUG [hconnection-0x79e1e112-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:53:44,629 DEBUG [hconnection-0x1b518f7a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:53:44,630 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50852, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:53:44,639 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50850, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:53:44,640 DEBUG [hconnection-0x3fdf621b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:53:44,679 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50862, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:53:44,681 DEBUG [hconnection-0xce38a73-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:53:44,681 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50880, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:53:44,685 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50874, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:53:44,686 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50910, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:53:44,693 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50896, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:53:44,695 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50924, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:53:44,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:44,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:44,714 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:44,714 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:44,716 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:44,716 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:44,721 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:44,723 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50928, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:53:44,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:44,730 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:44,730 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:44,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:44,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:44,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:44,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:44,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:44,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:44,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:44,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:44,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:44,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:44,740 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:44,740 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:44,743 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:44,749 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:44,749 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:44,753 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:44,753 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:44,756 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-27 03:53:44,768 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:44,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:44,778 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:53:44,778 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:44,779 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:53:44,779 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:44,779 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:53:44,780 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:44,906 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/290dc2b81b82472a93ec481e035ede24 2023-04-27 03:53:44,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682567624838, max timestamp=1682567624895Num uniques 859; 2023-04-27 03:53:44,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-27 03:53:44,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567624854, max timestamp=1682567624895Num uniques 859; 2023-04-27 03:53:44,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:53:44,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682567624854, max timestamp=1682567624895Num uniques 859; 2023-04-27 03:53:44,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-27 03:53:45,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:45,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567685025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:45,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567685028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:45,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567685030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:45,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567685034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:45,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50874 deadline: 1682567685037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,092 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/19ea049a37b04fc9ad421c77fe06e7e1 2023-04-27 03:53:45,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:45,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567685154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,173 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:45,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567685154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:45,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567685174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,180 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:45,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50874 deadline: 1682567685174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:45,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567685176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:45,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567685372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:45,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567685379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:45,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567685386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,393 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:45,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567685390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,394 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:45,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50874 deadline: 1682567685390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,653 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/01f0e954bd904a248996e50d5d9ede70 2023-04-27 03:53:45,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:45,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567685679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:45,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50874 deadline: 1682567685699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,708 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/290dc2b81b82472a93ec481e035ede24 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/290dc2b81b82472a93ec481e035ede24 2023-04-27 03:53:45,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:45,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567685700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:45,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567685705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:45,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567685701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,729 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/290dc2b81b82472a93ec481e035ede24, entries=150, sequenceid=11, filesize=11.6 K 2023-04-27 03:53:45,738 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/19ea049a37b04fc9ad421c77fe06e7e1 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/19ea049a37b04fc9ad421c77fe06e7e1 2023-04-27 03:53:45,776 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/19ea049a37b04fc9ad421c77fe06e7e1, entries=150, sequenceid=11, filesize=11.6 K 2023-04-27 03:53:45,786 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/01f0e954bd904a248996e50d5d9ede70 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/01f0e954bd904a248996e50d5d9ede70 2023-04-27 03:53:45,827 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/01f0e954bd904a248996e50d5d9ede70, entries=150, sequenceid=11, filesize=11.6 K 2023-04-27 03:53:45,830 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=136.02 KB/139285 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 1074ms, sequenceid=11, compaction requested=false 2023-04-27 03:53:45,832 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-27 03:53:45,838 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:45,838 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:45,838 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:45,839 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:45,839 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,839 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:45,839 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:45,847 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:45,847 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:45,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:45,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:45,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:45,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:45,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:45,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:45,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:45,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:45,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:45,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11c9a2a2[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:45,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:45,855 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:45,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:45,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:45,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:45,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:45,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:45,856 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,856 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:45,860 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:45,860 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:45,860 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:45,860 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:45,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:45,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:45,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:45,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:45,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:45,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:45,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:45,867 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:45,867 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:45,867 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:45,867 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:45,867 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:45,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:45,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:45,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:45,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:45,872 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:45,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:45,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:45,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:45,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:45,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:45,874 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,889 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,889 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:45,889 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:45,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:53:45,889 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:45,889 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:45,889 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:45,889 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:45,890 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:45,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:45,890 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:45,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:45,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:45,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:45,893 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:45,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:45,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:45,895 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:45,895 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:45,995 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:46,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:46,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:46,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:46,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:46,040 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:46,040 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:46,050 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:46,050 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:46,051 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:46,051 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:46,054 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:46,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:46,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:46,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:46,055 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:46,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:46,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:46,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:46,056 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:46,056 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:46,057 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-27 03:53:46,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:46,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:46,086 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:46,087 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:46,088 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:46,099 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:46,099 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:46,099 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:46,100 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:46,100 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=136.02 KB heapSize=384.46 KB 2023-04-27 03:53:46,100 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:53:46,101 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:53:46,101 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:53:46,101 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:53:46,101 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:53:46,102 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:53:46,222 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:46,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:46,292 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-27 03:53:46,293 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-27 03:53:46,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:46,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567686252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:46,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:46,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567686255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:46,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:46,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50874 deadline: 1682567686253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:46,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:46,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567686298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:46,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:46,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567686298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:46,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:46,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567686406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:46,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:46,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50874 deadline: 1682567686418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:46,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:46,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:46,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567686419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:46,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567686418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:46,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:46,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567686420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:46,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:46,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567686612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:46,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:46,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50874 deadline: 1682567686627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:46,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:46,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567686628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:46,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:46,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567686629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:46,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:46,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567686629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:46,698 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/07af5d9ed7e645929a99a4b91727aad3 2023-04-27 03:53:46,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:46,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567686922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:46,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:46,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567686931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:46,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:46,937 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:46,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50874 deadline: 1682567686932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:46,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567686932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:46,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:46,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567686936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:47,130 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/81d5388308124386bef5b4a92941edc2 2023-04-27 03:53:47,243 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-27 03:53:47,244 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-27 03:53:47,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:47,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567687429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:47,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:47,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50874 deadline: 1682567687441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:47,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:47,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567687443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:47,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:47,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567687444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:47,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:47,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567687451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:47,605 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/60616a58a61943a1a26872c232066fe7 2023-04-27 03:53:47,618 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/07af5d9ed7e645929a99a4b91727aad3 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/07af5d9ed7e645929a99a4b91727aad3 2023-04-27 03:53:47,632 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/07af5d9ed7e645929a99a4b91727aad3, entries=150, sequenceid=47, filesize=11.6 K 2023-04-27 03:53:47,636 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/81d5388308124386bef5b4a92941edc2 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/81d5388308124386bef5b4a92941edc2 2023-04-27 03:53:47,657 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/81d5388308124386bef5b4a92941edc2, entries=150, sequenceid=47, filesize=11.6 K 2023-04-27 03:53:47,662 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/60616a58a61943a1a26872c232066fe7 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/60616a58a61943a1a26872c232066fe7 2023-04-27 03:53:47,684 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/60616a58a61943a1a26872c232066fe7, entries=150, sequenceid=47, filesize=11.6 K 2023-04-27 03:53:47,703 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~136.02 KB/139285, heapSize ~384.41 KB/393640, currentSize=53.67 KB/54960 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 1603ms, sequenceid=47, compaction requested=false 2023-04-27 03:53:47,703 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:47,703 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:47,703 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:47,704 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:47,704 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:47,705 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:47,705 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:47,711 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:47,711 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:47,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:47,712 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:47,712 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:47,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:47,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:47,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5284adbd[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:47,716 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:47,716 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:47,717 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:47,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:47,717 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:47,717 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:47,717 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:47,718 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:47,718 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:47,722 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:47,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:47,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:47,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:47,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:47,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:47,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:47,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:47,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:47,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:47,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:47,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:47,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:47,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:47,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:47,729 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:47,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:47,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:47,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:47,729 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:47,742 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:47,742 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:47,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:47,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:47,743 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:47,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:47,747 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:47,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:47,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:47,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:47,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:47,750 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:47,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:47,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:47,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:47,757 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:47,757 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:47,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:47,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:47,757 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:47,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:47,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:47,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:47,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:47,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:47,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:47,758 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:47,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:47,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:47,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:47,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:47,760 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:47,760 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:47,860 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:47,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:47,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:47,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:47,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:47,867 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:47,867 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:47,869 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:47,869 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:47,869 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:47,869 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:47,871 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:47,871 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:47,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:47,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:47,872 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:47,872 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:47,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:47,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:47,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:47,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:47,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:47,874 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:47,874 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:47,875 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:47,875 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:47,875 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:47,875 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:47,875 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:47,875 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-27 03:53:47,876 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:53:47,876 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:47,876 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:53:47,876 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:47,876 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:53:47,876 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:47,907 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/9210223583d54ca89ffc748d6318154f 2023-04-27 03:53:48,000 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/aef355e335fa46e099cde443f2018f48 2023-04-27 03:53:48,447 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/92a04ba2acdb4c28a7e7824773590036 2023-04-27 03:53:48,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:48,448 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:48,462 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/9210223583d54ca89ffc748d6318154f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/9210223583d54ca89ffc748d6318154f 2023-04-27 03:53:48,471 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/9210223583d54ca89ffc748d6318154f, entries=150, sequenceid=58, filesize=11.6 K 2023-04-27 03:53:48,474 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/aef355e335fa46e099cde443f2018f48 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/aef355e335fa46e099cde443f2018f48 2023-04-27 03:53:48,491 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/aef355e335fa46e099cde443f2018f48, entries=150, sequenceid=58, filesize=11.6 K 2023-04-27 03:53:48,496 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/92a04ba2acdb4c28a7e7824773590036 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/92a04ba2acdb4c28a7e7824773590036 2023-04-27 03:53:48,516 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/92a04ba2acdb4c28a7e7824773590036, entries=150, sequenceid=58, filesize=11.6 K 2023-04-27 03:53:48,517 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=114.05 KB/116790 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 642ms, sequenceid=58, compaction requested=true 2023-04-27 03:53:48,517 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:48,517 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:48,517 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:48,522 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:48,522 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,522 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:48,522 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:48,527 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,527 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:48,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:48,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:48,528 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:48,528 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:48,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:48,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b8a018d[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:48,530 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:48,530 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:48,538 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:48,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:48,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:48,538 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:48,538 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:48,538 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,538 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:48,538 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:48,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1682567628470, max timestamp=1682567628531Num uniques 859; 2023-04-27 03:53:48,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-27 03:53:48,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:48,543 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:48,543 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:48,543 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=100.63 KB heapSize=287.50 KB 2023-04-27 03:53:48,543 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:48,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:48,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1682567628470, max timestamp=1682567628531Num uniques 859; 2023-04-27 03:53:48,543 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:48,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:48,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:48,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:48,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567628470, max timestamp=1682567628531Num uniques 859; 2023-04-27 03:53:48,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:53:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-27 03:53:48,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:48,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:48,548 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:48,548 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:48,548 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:48,548 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:48,548 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:48,548 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:53:48,549 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:53:48,549 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:53:48,549 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:53:48,549 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:53:48,549 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:53:48,550 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,550 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:48,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,550 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:48,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:48,551 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:48,554 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,570 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,570 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:48,570 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:48,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:53:48,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:48,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:48,570 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:48,570 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:48,570 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,570 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:48,570 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:48,570 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,570 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:48,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:48,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:48,572 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:48,572 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:48,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:48,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:48,594 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/4d9e7354066f4d9ebae58760ffc9dc9f 2023-04-27 03:53:48,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567628625, max timestamp=1682567628633Num uniques 859; 2023-04-27 03:53:48,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:53:48,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567628625, max timestamp=1682567628634Num uniques 859; 2023-04-27 03:53:48,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:53:48,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682567628625, max timestamp=1682567628633Num uniques 859; 2023-04-27 03:53:48,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-27 03:53:48,673 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:48,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:48,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:48,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:48,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:48,692 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:48,692 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:48,693 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:48,693 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:48,694 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,694 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:48,696 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:48,696 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:48,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:48,696 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,696 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:48,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:48,698 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:48,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:48,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:48,704 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:48,704 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:48,707 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:48,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:48,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567688690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:48,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:48,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567688694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50874 deadline: 1682567688695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,712 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:48,712 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:48,716 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:48,716 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:48,717 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:48,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:48,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567688710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,717 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:48,718 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:48,718 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:48,718 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,718 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:48,718 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:48,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:48,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567688712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,730 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:48,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:48,730 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:48,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:48,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:48,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:48,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:48,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:48,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:48,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:48,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55fcd3ca[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:48,734 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:48,734 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:48,736 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:48,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:48,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:48,736 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:48,736 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:48,736 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:48,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:48,739 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:48,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:48,740 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:48,740 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:48,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:48,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:48,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:48,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:48,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:48,748 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:48,748 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:48,748 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:48,749 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:48,749 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:48,751 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,751 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,751 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:48,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:48,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:48,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:48,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:48,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:48,752 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:48,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:48,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:48,755 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:48,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:48,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,772 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:48,772 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:48,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:48,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:48,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:48,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:48,775 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,775 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:48,775 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:48,775 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,775 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:48,775 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:48,776 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:48,776 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:48,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:48,781 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:48,781 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:48,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:53:48,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:48,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567688814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:48,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50874 deadline: 1682567688814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:48,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567688815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:48,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567688820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:48,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567688820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,881 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:48,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:48,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:48,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:48,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:48,892 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:48,892 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:48,897 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:48,897 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:48,897 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,897 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:48,899 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:48,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:48,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:48,899 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:48,900 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,900 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:48,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:48,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:48,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:48,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:48,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:48,901 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:48,901 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:48,902 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:48,902 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:48,902 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:48,902 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:48,902 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:48,903 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:48,903 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:48,903 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:48,903 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:48,903 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,903 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:48,903 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:48,906 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:48,906 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:48,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:48,907 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:48,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:48,907 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:48,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:48,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@97ca765[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:48,910 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:48,910 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:48,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:48,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:48,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:48,912 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:48,912 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:48,912 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:48,912 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,912 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:48,914 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,914 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:48,914 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:48,914 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:48,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:48,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:48,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:48,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:48,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:48,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:48,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:48,921 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:48,921 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:48,921 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:48,921 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:48,922 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:48,923 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,923 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,923 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:48,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:48,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:48,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:48,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:48,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:48,926 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:48,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:48,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:48,928 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,940 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:48,940 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,941 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:48,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:48,940 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:48,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:48,941 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:48,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:48,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:48,942 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:48,942 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:48,942 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:48,943 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:48,943 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:48,943 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:48,943 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:48,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:49,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:49,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567689019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:49,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567689024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:49,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50874 deadline: 1682567689024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:49,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567689029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:49,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567689031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,043 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:49,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:49,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:49,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:49,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:49,060 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:49,060 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:49,061 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:49,061 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:49,062 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,062 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:49,064 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/df00308aa3d14f859efca7649e969b71 2023-04-27 03:53:49,064 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,064 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:49,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:49,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:49,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,067 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,067 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:49,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:49,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:49,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:49,075 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:49,075 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:49,076 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:49,077 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:49,077 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:49,077 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:49,077 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:49,077 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:49,077 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:49,077 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:49,077 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:49,077 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,077 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:49,077 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:49,080 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:49,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:49,081 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:49,082 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,082 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:49,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:49,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:49,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:49,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ee21ab2[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:49,086 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:49,086 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,087 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,088 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:49,088 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:49,088 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:49,089 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,089 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:49,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:49,095 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:49,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:49,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:49,105 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:49,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:49,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:49,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:49,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:49,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:49,114 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/00c3f9aab5ad4126be46c240e93d4eb3 2023-04-27 03:53:49,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:49,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:49,115 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:49,115 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:49,116 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:49,116 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:49,116 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:49,122 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,122 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:49,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:49,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,122 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:49,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:49,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:49,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:49,125 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,126 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/4d9e7354066f4d9ebae58760ffc9dc9f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/4d9e7354066f4d9ebae58760ffc9dc9f 2023-04-27 03:53:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:49,129 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,135 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:49,135 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:49,135 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:49,135 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:49,135 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:49,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:49,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:49,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:53:49,135 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:49,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,137 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:49,137 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:49,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:49,139 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/4d9e7354066f4d9ebae58760ffc9dc9f, entries=300, sequenceid=81, filesize=18.5 K 2023-04-27 03:53:49,144 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/df00308aa3d14f859efca7649e969b71 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/df00308aa3d14f859efca7649e969b71 2023-04-27 03:53:49,158 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/df00308aa3d14f859efca7649e969b71, entries=150, sequenceid=81, filesize=11.6 K 2023-04-27 03:53:49,160 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/00c3f9aab5ad4126be46c240e93d4eb3 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/00c3f9aab5ad4126be46c240e93d4eb3 2023-04-27 03:53:49,176 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/00c3f9aab5ad4126be46c240e93d4eb3, entries=150, sequenceid=81, filesize=11.6 K 2023-04-27 03:53:49,179 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~62.62 KB/64120, heapSize ~156.81 KB/160576, currentSize=133.78 KB/136995 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 635ms, sequenceid=81, compaction requested=true 2023-04-27 03:53:49,181 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:49,183 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:53:49,184 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:53:49,184 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:53:49,184 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 03:53:49,184 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 03:53:49,189 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 03:53:49,189 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54564 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 03:53:49,191 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/B is initiating minor compaction (all files) 2023-04-27 03:53:49,191 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/A is initiating minor compaction (all files) 2023-04-27 03:53:49,191 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/B in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:49,191 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/A in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:49,191 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/19ea049a37b04fc9ad421c77fe06e7e1, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/81d5388308124386bef5b4a92941edc2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/aef355e335fa46e099cde443f2018f48, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/df00308aa3d14f859efca7649e969b71] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=46.4 K 2023-04-27 03:53:49,191 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/290dc2b81b82472a93ec481e035ede24, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/07af5d9ed7e645929a99a4b91727aad3, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/9210223583d54ca89ffc748d6318154f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/4d9e7354066f4d9ebae58760ffc9dc9f] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=53.3 K 2023-04-27 03:53:49,192 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 19ea049a37b04fc9ad421c77fe06e7e1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1682567624728 2023-04-27 03:53:49,195 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 290dc2b81b82472a93ec481e035ede24, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1682567624728 2023-04-27 03:53:49,196 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 81d5388308124386bef5b4a92941edc2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1682567625026 2023-04-27 03:53:49,196 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 07af5d9ed7e645929a99a4b91727aad3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1682567625026 2023-04-27 03:53:49,197 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 9210223583d54ca89ffc748d6318154f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682567626222 2023-04-27 03:53:49,197 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting aef355e335fa46e099cde443f2018f48, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682567626222 2023-04-27 03:53:49,198 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting df00308aa3d14f859efca7649e969b71, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1682567628470 2023-04-27 03:53:49,198 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 4d9e7354066f4d9ebae58760ffc9dc9f, keycount=300, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1682567628470 2023-04-27 03:53:49,237 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:49,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:49,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:49,278 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#B#compaction#12 average throughput is 0.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:53:49,279 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#A#compaction#13 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:53:49,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:49,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:49,281 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:49,281 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:49,282 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:49,282 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:49,290 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,290 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:49,292 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:49,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:49,292 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:49,293 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,293 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:49,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:49,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:49,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:49,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:49,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:49,309 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:49,309 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:49,310 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:49,344 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:49,344 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:49,344 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:49,344 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:49,344 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=140.49 KB heapSize=404.70 KB 2023-04-27 03:53:49,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682567628690, max timestamp=1682567629334Num uniques 1018; 2023-04-27 03:53:49,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-27 03:53:49,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682567628690, max timestamp=1682567629340Num uniques 1009; 2023-04-27 03:53:49,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-27 03:53:49,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:49,355 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:49,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682567628690, max timestamp=1682567629329Num uniques 1018; 2023-04-27 03:53:49,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-27 03:53:49,368 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:53:49,368 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:53:49,368 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:53:49,368 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:53:49,369 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:53:49,369 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:53:49,424 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/fc259a044e954d0bb8febb6e76c741e9 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/fc259a044e954d0bb8febb6e76c741e9 2023-04-27 03:53:49,426 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/8fb41cd0c8584d969865e91211672c6b 2023-04-27 03:53:49,462 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/A of 2b04f11bbacd80b8750d75eb2d0fbb35 into fc259a044e954d0bb8febb6e76c741e9(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:53:49,462 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:49,462 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/A, priority=12, startTime=1682567629183; duration=0sec 2023-04-27 03:53:49,465 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:53:49,465 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 03:53:49,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682567629444, max timestamp=1682567629464Num uniques 859; 2023-04-27 03:53:49,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-27 03:53:49,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682567629444, max timestamp=1682567629464Num uniques 859; 2023-04-27 03:53:49,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-27 03:53:49,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682567629444, max timestamp=1682567629464Num uniques 859; 2023-04-27 03:53:49,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-27 03:53:49,469 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 03:53:49,470 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/C is initiating minor compaction (all files) 2023-04-27 03:53:49,470 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/C in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:49,470 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/01f0e954bd904a248996e50d5d9ede70, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/60616a58a61943a1a26872c232066fe7, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/92a04ba2acdb4c28a7e7824773590036, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/00c3f9aab5ad4126be46c240e93d4eb3] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=46.4 K 2023-04-27 03:53:49,471 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 01f0e954bd904a248996e50d5d9ede70, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1682567624728 2023-04-27 03:53:49,471 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 60616a58a61943a1a26872c232066fe7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1682567625026 2023-04-27 03:53:49,472 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 92a04ba2acdb4c28a7e7824773590036, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682567626222 2023-04-27 03:53:49,473 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 00c3f9aab5ad4126be46c240e93d4eb3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1682567628470 2023-04-27 03:53:49,484 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/9c90a21c05664d53b1c4d23f3a4694d7 2023-04-27 03:53:49,502 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#C#compaction#17 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:53:49,529 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/089e53dc16e54371b6a716f071b85efe 2023-04-27 03:53:49,539 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/1c1955c6a031432f9b30827f89e3d060 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/1c1955c6a031432f9b30827f89e3d060 2023-04-27 03:53:49,540 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/8fb41cd0c8584d969865e91211672c6b as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/8fb41cd0c8584d969865e91211672c6b 2023-04-27 03:53:49,554 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/8fb41cd0c8584d969865e91211672c6b, entries=150, sequenceid=121, filesize=11.6 K 2023-04-27 03:53:49,556 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/9c90a21c05664d53b1c4d23f3a4694d7 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/9c90a21c05664d53b1c4d23f3a4694d7 2023-04-27 03:53:49,568 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/9c90a21c05664d53b1c4d23f3a4694d7, entries=150, sequenceid=121, filesize=11.6 K 2023-04-27 03:53:49,569 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/089e53dc16e54371b6a716f071b85efe as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/089e53dc16e54371b6a716f071b85efe 2023-04-27 03:53:49,571 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/C of 2b04f11bbacd80b8750d75eb2d0fbb35 into 1c1955c6a031432f9b30827f89e3d060(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:53:49,572 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:49,572 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/C, priority=12, startTime=1682567629184; duration=0sec 2023-04-27 03:53:49,573 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:53:49,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:49,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50874 deadline: 1682567689515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:49,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567689517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,611 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/089e53dc16e54371b6a716f071b85efe, entries=150, sequenceid=121, filesize=11.6 K 2023-04-27 03:53:49,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:49,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567689584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:49,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567689584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:49,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567689589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,622 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~105.19 KB/107712, currentSize=147.60 KB/151140 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 277ms, sequenceid=121, compaction requested=false 2023-04-27 03:53:49,623 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:49,623 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:49,623 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:49,623 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:49,623 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,623 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:49,623 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:49,625 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:49,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:49,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:49,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:49,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:49,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:49,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:49,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:49,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@756fea8a[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:49,629 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:49,629 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,632 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:49,632 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:49,632 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:49,632 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,633 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:49,633 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:49,633 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,633 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:49,634 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:49,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:49,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:49,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:49,639 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:49,640 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:49,640 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:49,640 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:49,640 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:49,641 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,642 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:49,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,642 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:49,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:49,643 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:49,645 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,649 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,649 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:49,650 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:49,650 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:49,650 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,650 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,650 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:49,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:49,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:49,650 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:49,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:49,650 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,653 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,653 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:49,653 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:49,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:49,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:49,719 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=154.31 KB heapSize=436.08 KB 2023-04-27 03:53:49,719 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:53:49,719 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-27 03:53:49,719 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:53:49,719 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-27 03:53:49,719 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:53:49,720 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-27 03:53:49,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:49,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567689725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:49,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567689726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:49,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567689725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:49,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567689730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:49,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50874 deadline: 1682567689732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,754 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:49,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:49,759 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/e35a73efddf54835936ca56e544ec084 2023-04-27 03:53:49,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:49,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:49,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:49,777 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:49,777 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:49,778 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:49,778 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:49,785 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,785 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:49,787 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:49,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:49,787 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:49,788 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,788 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:49,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:49,790 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:49,791 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/10895b2bf00e43c4bc136a8dcfe373f2 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/10895b2bf00e43c4bc136a8dcfe373f2 2023-04-27 03:53:49,805 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/B of 2b04f11bbacd80b8750d75eb2d0fbb35 into 10895b2bf00e43c4bc136a8dcfe373f2(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:53:49,805 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:49,805 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/B, priority=12, startTime=1682567629184; duration=0sec 2023-04-27 03:53:49,806 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:53:49,809 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:49,809 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:49,809 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:49,809 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:49,810 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:49,813 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:49,813 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:49,813 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:49,814 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:49,814 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:49,814 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:49,814 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:49,814 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:49,814 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,814 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:49,814 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:49,818 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:49,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:49,819 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:49,819 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,819 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:49,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:49,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:49,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:49,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:49,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@463e156e[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:49,821 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:49,821 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,827 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,827 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,827 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:49,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,827 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:49,827 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:49,827 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:49,829 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,829 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:49,830 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:49,830 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:49,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:49,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:49,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:49,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:49,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567689834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:49,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:49,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567689834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:49,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:49,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567689835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,841 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:49,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567689841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:49,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:49,843 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:49,843 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:49,843 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:49,843 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:49,843 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:49,845 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,845 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:49,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:49,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,845 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:49,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:49,846 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:49,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,849 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:49,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50874 deadline: 1682567689850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,856 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:49,856 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:49,856 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:49,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:49,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:49,857 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:49,857 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:49,856 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:49,858 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:49,858 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,858 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,858 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,858 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,856 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:49,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:49,958 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:49,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:49,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:49,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:49,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:49,964 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:49,965 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:49,966 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:49,966 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:49,967 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,967 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:49,968 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:49,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:49,968 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:49,969 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,969 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:49,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:49,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:49,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:49,972 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:49,972 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:49,972 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:49,975 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:49,975 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:49,976 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:49,976 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:49,976 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:49,976 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:49,977 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:49,977 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:49,977 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:49,977 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:49,977 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:49,977 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,977 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:49,977 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:49,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,979 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:49,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:49,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:49,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a15c543[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:49,983 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:49,983 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,985 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:49,985 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,985 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:49,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:49,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:49,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:49,989 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:49,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:49,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:49,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:49,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:49,995 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:49,995 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:49,995 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:49,995 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:49,995 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:49,997 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,997 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:49,997 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:49,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:49,999 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:50,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,001 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:50,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:50,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:50,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:50,006 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:50,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:50,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:50,008 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:50,008 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:50,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:53:50,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:50,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567690039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:50,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567690039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:50,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567690039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:50,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567690044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:50,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50874 deadline: 1682567690052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,108 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:50,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:50,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:50,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:50,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:50,114 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:50,114 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:50,115 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:50,115 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:50,116 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,116 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,118 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,118 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:50,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:50,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:50,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,119 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,119 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:50,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:50,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:50,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:50,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:50,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:50,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:50,126 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:50,126 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:50,126 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:50,127 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:50,127 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:50,127 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:50,127 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:50,127 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:50,127 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:50,127 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,127 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:50,127 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:50,130 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,130 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:50,131 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,131 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:50,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:50,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:50,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44cc6d39[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:50,133 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:50,133 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,134 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,134 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:50,134 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:50,134 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:50,134 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,134 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:50,136 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:50,136 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,136 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:50,136 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:50,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:50,142 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:50,142 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:50,142 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:50,142 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:50,142 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:50,144 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,144 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:50,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,145 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,146 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,148 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:50,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,151 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,151 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,151 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,151 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:50,151 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:50,151 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:50,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:53:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:50,151 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,152 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,152 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,151 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:50,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:50,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:50,152 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:50,152 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:50,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:50,226 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/2861b217e13a4a9688a18b70d919c321 2023-04-27 03:53:50,253 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:50,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:50,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:50,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:50,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:50,264 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:50,264 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:50,265 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:50,265 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:50,266 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,266 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,267 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/16609fe422bc44379a15b149b59487ec 2023-04-27 03:53:50,268 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,268 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:50,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:50,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:50,269 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,269 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:50,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:50,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:50,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:50,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:50,271 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:50,271 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:50,272 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:50,278 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/e35a73efddf54835936ca56e544ec084 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/e35a73efddf54835936ca56e544ec084 2023-04-27 03:53:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:50,280 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:50,280 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:50,280 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:50,280 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:50,281 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,281 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:50,281 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:50,283 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,283 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:50,284 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,284 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:50,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:50,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e7de490[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:50,287 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:50,288 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,288 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/e35a73efddf54835936ca56e544ec084, entries=200, sequenceid=161, filesize=14.1 K 2023-04-27 03:53:50,289 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,289 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:50,289 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:50,289 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:50,289 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,289 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:50,292 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/2861b217e13a4a9688a18b70d919c321 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/2861b217e13a4a9688a18b70d919c321 2023-04-27 03:53:50,297 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,297 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:50,297 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:50,298 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:50,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:50,300 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:50,300 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:50,300 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:50,300 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:50,300 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:50,302 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,302 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,302 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:50,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:50,303 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,307 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,311 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:50,311 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:50,311 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,311 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:50,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:50,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:50,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:50,311 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,313 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,313 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,313 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,313 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,313 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:50,313 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:50,320 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:50,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:50,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:50,321 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/2861b217e13a4a9688a18b70d919c321, entries=150, sequenceid=161, filesize=11.8 K 2023-04-27 03:53:50,323 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/16609fe422bc44379a15b149b59487ec as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/16609fe422bc44379a15b149b59487ec 2023-04-27 03:53:50,339 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/16609fe422bc44379a15b149b59487ec, entries=150, sequenceid=161, filesize=11.8 K 2023-04-27 03:53:50,345 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~436.03 KB/446496, currentSize=33.54 KB/34350 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 626ms, sequenceid=161, compaction requested=true 2023-04-27 03:53:50,345 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:50,345 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:53:50,345 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:53:50,345 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:53:50,346 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:53:50,346 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:53:50,349 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38336 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:53:50,350 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/A is initiating minor compaction (all files) 2023-04-27 03:53:50,350 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/A in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:50,350 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/fc259a044e954d0bb8febb6e76c741e9, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/8fb41cd0c8584d969865e91211672c6b, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/e35a73efddf54835936ca56e544ec084] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=37.4 K 2023-04-27 03:53:50,349 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35946 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:53:50,350 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/B is initiating minor compaction (all files) 2023-04-27 03:53:50,350 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/B in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:50,351 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/10895b2bf00e43c4bc136a8dcfe373f2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/9c90a21c05664d53b1c4d23f3a4694d7, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/2861b217e13a4a9688a18b70d919c321] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=35.1 K 2023-04-27 03:53:50,351 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting fc259a044e954d0bb8febb6e76c741e9, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1682567628470 2023-04-27 03:53:50,353 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 10895b2bf00e43c4bc136a8dcfe373f2, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1682567628470 2023-04-27 03:53:50,354 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 8fb41cd0c8584d969865e91211672c6b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1682567628708 2023-04-27 03:53:50,355 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 9c90a21c05664d53b1c4d23f3a4694d7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1682567628708 2023-04-27 03:53:50,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:50,355 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting e35a73efddf54835936ca56e544ec084, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682567629513 2023-04-27 03:53:50,355 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 2861b217e13a4a9688a18b70d919c321, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682567629515 2023-04-27 03:53:50,356 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:53:50,360 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:53:50,361 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:50,361 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:53:50,361 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:50,361 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:53:50,361 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:50,379 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#A#compaction#22 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:53:50,404 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#B#compaction#23 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:53:50,416 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:50,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:50,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:50,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:50,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:50,447 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:50,447 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:50,448 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:50,448 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:50,449 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,449 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,451 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:50,451 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:50,451 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,451 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:50,454 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:50,454 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:50,454 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:50,454 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:50,455 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:50,455 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:50,455 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:50,455 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:50,455 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:50,455 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:50,456 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:50,456 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:50,456 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:50,456 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,456 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:50,456 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:50,462 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:50,462 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:50,463 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,463 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:50,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:50,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78de0d54[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:50,464 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:50,465 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,466 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:50,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:50,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:50,467 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,468 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:50,469 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:50,469 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:50,469 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:50,469 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:50,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,475 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/4c437e0b86e94225b34e839c9976866a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/4c437e0b86e94225b34e839c9976866a 2023-04-27 03:53:50,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:50,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:50,478 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:50,478 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:50,478 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:50,478 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:50,478 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:50,481 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:50,481 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,481 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:50,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:50,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:50,482 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,486 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,493 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:50,493 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:50,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:50,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:50,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:50,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:50,494 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,494 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:50,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:50,494 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,494 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:50,495 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,495 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:50,501 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:50,495 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,502 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,502 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,502 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/A of 2b04f11bbacd80b8750d75eb2d0fbb35 into 4c437e0b86e94225b34e839c9976866a(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:53:50,502 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:50,502 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/A, priority=13, startTime=1682567630345; duration=0sec 2023-04-27 03:53:50,502 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:53:50,502 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:53:50,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567630486, max timestamp=1682567630500Num uniques 859; 2023-04-27 03:53:50,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:53:50,508 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35946 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:53:50,509 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/C is initiating minor compaction (all files) 2023-04-27 03:53:50,509 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/C in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:50,509 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/1c1955c6a031432f9b30827f89e3d060, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/089e53dc16e54371b6a716f071b85efe, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/16609fe422bc44379a15b149b59487ec] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=35.1 K 2023-04-27 03:53:50,515 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 1c1955c6a031432f9b30827f89e3d060, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1682567628470 2023-04-27 03:53:50,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:50,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567690506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:50,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567690509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:50,517 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 089e53dc16e54371b6a716f071b85efe, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1682567628708 2023-04-27 03:53:50,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567690511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:50,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567690512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,519 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 16609fe422bc44379a15b149b59487ec, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682567629515 2023-04-27 03:53:50,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:50,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50874 deadline: 1682567690515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,532 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#C#compaction#24 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:53:50,566 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/523c68810fea48a280fd571c0cfe79ce as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/523c68810fea48a280fd571c0cfe79ce 2023-04-27 03:53:50,576 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/C of 2b04f11bbacd80b8750d75eb2d0fbb35 into 523c68810fea48a280fd571c0cfe79ce(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:53:50,576 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:50,577 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/C, priority=13, startTime=1682567630346; duration=0sec 2023-04-27 03:53:50,577 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:53:50,602 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:50,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:50,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:50,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:50,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:50,607 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:50,607 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:50,609 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:50,609 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:50,609 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,609 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,611 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:50,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:50,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:50,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:50,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:50,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:50,613 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:50,613 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:50,614 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:50,614 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,614 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:50,614 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:50,615 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:50,615 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:50,615 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:50,615 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:50,615 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:50,615 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:50,615 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,615 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:50,615 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:50,615 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,615 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:50,619 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:50,619 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:50,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:50,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:50,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567690618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:50,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:50,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ce5f0d1[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:50,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:50,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567690620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:50,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567690620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,625 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:50,625 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:50,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567690620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,625 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,626 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:50,627 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:50,627 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:50,627 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,627 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:50,630 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,630 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:50,630 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:50,630 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:50,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:50,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50874 deadline: 1682567690624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:50,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:50,635 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:50,635 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:50,635 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:50,635 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:50,635 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:50,636 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,637 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,637 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:50,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,638 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,639 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,642 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,642 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,642 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,642 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,642 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,642 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:50,642 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:50,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:50,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:50,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:53:50,642 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,644 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:50,643 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:50,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:50,644 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:50,644 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:50,744 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:50,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:50,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:50,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:50,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:50,756 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:50,756 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:50,757 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:50,757 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:50,758 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,758 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,759 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:50,759 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:50,760 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,760 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:50,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:50,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:50,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:50,761 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:50,761 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:50,762 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:50,763 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:50,763 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:50,763 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:50,763 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:50,763 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:50,763 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:50,763 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:50,763 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:50,763 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,763 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:50,763 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:50,766 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,766 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:50,766 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,766 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:50,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:50,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b67efd1[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:50,768 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:50,769 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,770 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,770 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:50,770 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:50,770 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:50,770 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,771 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:50,772 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:50,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:50,776 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:50,776 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:50,777 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:50,777 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:50,777 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:50,779 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,779 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:50,779 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:50,780 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,783 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:50,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:50,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,788 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:50,788 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:50,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:50,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:50,789 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,789 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,789 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,789 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,789 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,789 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,790 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,790 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:50,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:50,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:50,793 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:50,793 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:50,825 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:50,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567690824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:50,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567690826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:50,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567690827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:50,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567690827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:50,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50874 deadline: 1682567690834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,850 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/520cd0056c744611b6afe8a028c811bf 2023-04-27 03:53:50,868 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/a6fc24ae4b544892a9c82fcd498ca4f5 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/a6fc24ae4b544892a9c82fcd498ca4f5 2023-04-27 03:53:50,880 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/B of 2b04f11bbacd80b8750d75eb2d0fbb35 into a6fc24ae4b544892a9c82fcd498ca4f5(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:53:50,880 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:50,880 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/B, priority=13, startTime=1682567630345; duration=0sec 2023-04-27 03:53:50,880 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:53:50,893 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:50,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:50,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:50,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:50,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:50,904 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:50,904 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:50,909 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:50,909 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:50,909 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,909 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,912 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:50,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:50,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:50,914 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,920 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,920 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:50,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:50,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:50,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:50,922 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:50,922 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:50,924 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:50,925 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:50,925 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:50,925 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:50,925 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:50,925 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:50,925 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:50,925 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:50,925 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:50,925 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,925 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:50,925 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:50,927 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:50,928 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/862a687c448f4ba1b8345402a92481bb 2023-04-27 03:53:50,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:50,929 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:50,933 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,933 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:50,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:50,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a1fc949[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:50,935 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:50,935 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,941 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,941 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:50,941 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:50,941 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:50,942 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,942 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:50,943 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:50,943 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:50,943 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:50,943 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:50,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:50,964 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:50,964 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:50,964 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:50,964 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:50,964 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:50,966 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,966 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,966 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:50,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:50,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:50,967 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,970 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:50,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:50,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,974 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:50,975 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:50,975 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:50,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:50,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:50,975 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,975 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:50,975 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:50,975 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,975 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:50,975 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:50,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:50,975 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:50,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:50,993 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:50,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:51,014 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/1844e6156bce4dd995ea97da046f6bcd 2023-04-27 03:53:51,023 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/520cd0056c744611b6afe8a028c811bf as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/520cd0056c744611b6afe8a028c811bf 2023-04-27 03:53:51,033 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/520cd0056c744611b6afe8a028c811bf, entries=150, sequenceid=173, filesize=11.8 K 2023-04-27 03:53:51,036 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/862a687c448f4ba1b8345402a92481bb as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/862a687c448f4ba1b8345402a92481bb 2023-04-27 03:53:51,052 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/862a687c448f4ba1b8345402a92481bb, entries=150, sequenceid=173, filesize=11.8 K 2023-04-27 03:53:51,058 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/1844e6156bce4dd995ea97da046f6bcd as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/1844e6156bce4dd995ea97da046f6bcd 2023-04-27 03:53:51,068 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/1844e6156bce4dd995ea97da046f6bcd, entries=150, sequenceid=173, filesize=11.8 K 2023-04-27 03:53:51,070 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=123.00 KB/125950 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 714ms, sequenceid=173, compaction requested=false 2023-04-27 03:53:51,070 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:51,093 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:51,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:51,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:51,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:51,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:51,099 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:51,099 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:51,100 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:51,100 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:51,100 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,100 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:51,103 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:51,103 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:51,104 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,104 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:51,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:51,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:51,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:51,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:51,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:51,105 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:51,105 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:51,106 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:51,106 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:51,107 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:51,107 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:51,107 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:51,107 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=123.00 KB heapSize=361.72 KB 2023-04-27 03:53:51,107 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:53:51,107 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:53:51,107 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:53:51,108 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:53:51,108 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:53:51,108 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:53:51,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:51,130 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:51,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:51,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567691136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:51,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567691137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:51,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567691138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:51,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567691139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:51,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50874 deadline: 1682567691141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:51,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567691241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,246 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:51,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567691241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:51,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567691243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,250 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:51,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567691248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:51,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567691446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:51,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567691449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:51,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567691449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:51,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567691452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,528 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/62244adb83cb442c9fd3c8d5f6af0774 2023-04-27 03:53:51,554 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/00fb19af768f46ec92650aeb523b559f 2023-04-27 03:53:51,573 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/9d8c985afa814970be62721548859585 2023-04-27 03:53:51,581 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/62244adb83cb442c9fd3c8d5f6af0774 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/62244adb83cb442c9fd3c8d5f6af0774 2023-04-27 03:53:51,588 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/62244adb83cb442c9fd3c8d5f6af0774, entries=150, sequenceid=202, filesize=11.8 K 2023-04-27 03:53:51,589 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/00fb19af768f46ec92650aeb523b559f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/00fb19af768f46ec92650aeb523b559f 2023-04-27 03:53:51,602 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/00fb19af768f46ec92650aeb523b559f, entries=150, sequenceid=202, filesize=11.8 K 2023-04-27 03:53:51,605 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/9d8c985afa814970be62721548859585 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/9d8c985afa814970be62721548859585 2023-04-27 03:53:51,614 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/9d8c985afa814970be62721548859585, entries=150, sequenceid=202, filesize=11.8 K 2023-04-27 03:53:51,617 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~123.00 KB/125950, heapSize ~361.67 KB/370352, currentSize=53.67 KB/54960 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 510ms, sequenceid=202, compaction requested=true 2023-04-27 03:53:51,617 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:51,619 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:51,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:51,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:51,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:51,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:51,624 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:51,624 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:51,624 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:51,624 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:51,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:51,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72a3cff3[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:51,626 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:51,626 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:51,627 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:51,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:51,629 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,629 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:51,636 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:51,636 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:51,636 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:51,637 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,637 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:51,638 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:51,638 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:51,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:51,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:51,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:51,643 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:51,643 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:51,643 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:51,643 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:51,643 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:51,646 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:51,646 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,646 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:51,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:51,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:51,647 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:51,648 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:51,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:51,650 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-27 03:53:51,650 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:53:51,650 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:51,650 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:53:51,650 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:51,650 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:53:51,650 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,653 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,653 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:51,653 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:51,653 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,653 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:51,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:51,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:51,654 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:51,654 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:51,653 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:51,655 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:51,655 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,654 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:51,655 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:51,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:51,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:53:51,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:51,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:51,671 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/1f9915701d0348568310f503bb4cb204 2023-04-27 03:53:51,703 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/81a647d6d2a047d1ba63790e2c090064 2023-04-27 03:53:51,735 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/3466f481f3844a8eb4eb68e4325e65f4 2023-04-27 03:53:51,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:51,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50874 deadline: 1682567691733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,746 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/1f9915701d0348568310f503bb4cb204 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/1f9915701d0348568310f503bb4cb204 2023-04-27 03:53:51,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:51,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567691751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,754 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:51,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:51,756 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/1f9915701d0348568310f503bb4cb204, entries=150, sequenceid=214, filesize=11.8 K 2023-04-27 03:53:51,758 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/81a647d6d2a047d1ba63790e2c090064 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/81a647d6d2a047d1ba63790e2c090064 2023-04-27 03:53:51,759 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:51,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567691755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:51,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:51,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:51,762 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:51,763 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:51,765 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:51,765 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:51,765 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,765 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:51,767 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:51,767 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:51,767 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,767 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:51,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:51,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:51,769 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:51,769 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:51,770 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:51,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567691758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:51,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567691760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,779 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:51,779 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:51,784 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/81a647d6d2a047d1ba63790e2c090064, entries=150, sequenceid=214, filesize=11.8 K 2023-04-27 03:53:51,789 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:51,789 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:51,790 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:51,790 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:51,790 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:51,790 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:51,790 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,790 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:51,790 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:51,791 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/3466f481f3844a8eb4eb68e4325e65f4 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/3466f481f3844a8eb4eb68e4325e65f4 2023-04-27 03:53:51,792 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,792 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:51,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:51,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:51,792 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:51,792 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:51,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:51,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:51,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2113ca0e[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:51,795 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:51,795 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:51,797 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:51,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:51,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:51,797 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:51,797 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:51,797 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:51,797 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,797 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:51,799 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/3466f481f3844a8eb4eb68e4325e65f4, entries=150, sequenceid=214, filesize=11.8 K 2023-04-27 03:53:51,799 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,799 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,799 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:51,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:51,799 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:51,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:51,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:51,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:51,806 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:51,806 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:51,806 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:51,806 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:51,806 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:51,810 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,810 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:51,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,810 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:51,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:51,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:51,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,811 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:51,813 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,818 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 168ms, sequenceid=214, compaction requested=true 2023-04-27 03:53:51,818 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:51,818 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:53:51,818 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 03:53:51,818 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 03:53:51,818 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:53:51,818 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:53:51,819 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:51,819 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:51,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:51,820 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,820 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:51,820 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:51,820 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,820 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:51,820 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:51,820 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,820 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:51,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:53:51,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:51,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:51,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:51,822 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:51,822 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:51,823 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48384 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 03:53:51,823 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/A is initiating minor compaction (all files) 2023-04-27 03:53:51,823 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/A in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:51,824 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/4c437e0b86e94225b34e839c9976866a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/520cd0056c744611b6afe8a028c811bf, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/62244adb83cb442c9fd3c8d5f6af0774, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/1f9915701d0348568310f503bb4cb204] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=47.3 K 2023-04-27 03:53:51,824 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48384 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 03:53:51,824 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/B is initiating minor compaction (all files) 2023-04-27 03:53:51,824 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/B in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:51,824 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/a6fc24ae4b544892a9c82fcd498ca4f5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/862a687c448f4ba1b8345402a92481bb, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/00fb19af768f46ec92650aeb523b559f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/81a647d6d2a047d1ba63790e2c090064] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=47.3 K 2023-04-27 03:53:51,825 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 4c437e0b86e94225b34e839c9976866a, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682567629515 2023-04-27 03:53:51,826 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting a6fc24ae4b544892a9c82fcd498ca4f5, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682567629515 2023-04-27 03:53:51,828 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 520cd0056c744611b6afe8a028c811bf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1682567629724 2023-04-27 03:53:51,828 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 62244adb83cb442c9fd3c8d5f6af0774, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1682567630508 2023-04-27 03:53:51,828 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 862a687c448f4ba1b8345402a92481bb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1682567629724 2023-04-27 03:53:51,829 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 1f9915701d0348568310f503bb4cb204, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1682567631133 2023-04-27 03:53:51,830 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 00fb19af768f46ec92650aeb523b559f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1682567630508 2023-04-27 03:53:51,831 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 81a647d6d2a047d1ba63790e2c090064, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1682567631133 2023-04-27 03:53:51,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567631721, max timestamp=1682567631841Num uniques 859; 2023-04-27 03:53:51,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567631721, max timestamp=1682567631841Num uniques 859; 2023-04-27 03:53:51,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:53:51,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567631721, max timestamp=1682567631841Num uniques 859; 2023-04-27 03:53:51,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:51,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:53:51,844 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=26.84 KB heapSize=66.23 KB 2023-04-27 03:53:51,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:53:51,845 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:53:51,845 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:53:51,845 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:53:51,845 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:53:51,845 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:53:51,845 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:53:51,856 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#A#compaction#34 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:53:51,862 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#B#compaction#35 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:53:51,892 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/92d8f0f012504a9ab9bd1b2778ee168e 2023-04-27 03:53:51,911 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/3d67f32ca8f94546abad605f17dc4f77 2023-04-27 03:53:51,922 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:51,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:51,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:51,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:51,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:51,928 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:51,928 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:51,929 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:51,929 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:51,930 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,930 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:51,932 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,932 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:51,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:51,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:51,933 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,933 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:51,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:51,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:51,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:51,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:51,934 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:51,934 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:51,936 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:51,937 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:51,937 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:51,937 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:51,937 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:51,939 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,939 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:51,940 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:51,940 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:51,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:51,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45eeb09a[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:51,942 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:51,942 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:51,943 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:51,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:51,943 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:51,944 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:51,944 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:51,944 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,944 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:51,944 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/7470ae35249c4594867dfdba4ba1a4a5 2023-04-27 03:53:51,948 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:51,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:51,948 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:51,949 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:51,949 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:51,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567631912, max timestamp=1682567631947Num uniques 859; 2023-04-27 03:53:51,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:53:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:51,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:51,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:51,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567631912, max timestamp=1682567631947Num uniques 859; 2023-04-27 03:53:51,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:53:51,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567631912, max timestamp=1682567631947Num uniques 859; 2023-04-27 03:53:51,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:53:51,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:51,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:51,953 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:51,953 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:51,953 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:51,953 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:51,953 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:51,957 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/92d8f0f012504a9ab9bd1b2778ee168e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/92d8f0f012504a9ab9bd1b2778ee168e 2023-04-27 03:53:51,962 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,962 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:51,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,962 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:51,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:51,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:51,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:51,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:51,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:51,964 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,964 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/92d8f0f012504a9ab9bd1b2778ee168e, entries=200, sequenceid=235, filesize=14.1 K 2023-04-27 03:53:51,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:51,966 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/3d67f32ca8f94546abad605f17dc4f77 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/3d67f32ca8f94546abad605f17dc4f77 2023-04-27 03:53:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,967 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:51,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:51,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,970 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,971 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:51,971 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:51,971 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:51,971 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:51,971 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:51,971 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:51,971 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:51,971 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:51,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:51,971 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:51,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:51,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:51,973 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:51,973 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:51,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:51,975 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/3d67f32ca8f94546abad605f17dc4f77, entries=150, sequenceid=235, filesize=11.8 K 2023-04-27 03:53:51,979 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/7470ae35249c4594867dfdba4ba1a4a5 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/7470ae35249c4594867dfdba4ba1a4a5 2023-04-27 03:53:51,987 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/7470ae35249c4594867dfdba4ba1a4a5, entries=150, sequenceid=235, filesize=11.8 K 2023-04-27 03:53:51,988 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.19 KB/67776, currentSize=73.80 KB/75570 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 144ms, sequenceid=235, compaction requested=true 2023-04-27 03:53:51,988 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:51,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:51,988 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-27 03:53:51,988 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-27 03:53:51,988 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-27 03:53:51,989 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=80.51 KB heapSize=220.92 KB 2023-04-27 03:53:51,989 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:53:51,989 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:53:51,989 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:53:51,989 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:53:51,989 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:53:51,989 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:53:52,032 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/2b064e1fef164267a9fe4bf5e76485c2 2023-04-27 03:53:52,073 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:52,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:52,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:52,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:52,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:52,090 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:52,090 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:52,092 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:52,092 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:52,094 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,094 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:52,095 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,095 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:52,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:52,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:52,096 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,096 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:52,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:52,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:52,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:52,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:52,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:52,102 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:52,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567692093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:52,102 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:52,102 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:52,106 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:52,106 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:52,107 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:52,107 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:52,107 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:52,107 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:52,107 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:52,107 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:52,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:52,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:52,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:52,109 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:52,110 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:52,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:52,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:52,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:52,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:52,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:52,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41bf7cef[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:52,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:52,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:52,116 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:52,116 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:52,116 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:52,116 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:52,116 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:52,116 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:52,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:52,118 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:52,118 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,118 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:52,118 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:52,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:52,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:52,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:52,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:52,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:52,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:52,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:52,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:52,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:52,129 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:52,129 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:52,129 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:52,129 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:52,129 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:52,131 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,132 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:52,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,132 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:52,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:52,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:52,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:52,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:52,133 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:52,135 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:52,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:52,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,138 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,138 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:52,138 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:52,138 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:52,138 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:52,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:52,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:52,138 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:52,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:52,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:53:52,138 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:52,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:52,138 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,140 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:52,140 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:52,140 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:52,140 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:52,140 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:52,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:52,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567692208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,240 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:52,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:52,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:52,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:52,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:52,245 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:52,245 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:52,246 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:52,246 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:52,247 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,247 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:52,249 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,249 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:52,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:52,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:52,249 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,249 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:52,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:52,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:52,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:52,250 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:52,251 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:52,251 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:52,251 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:52,251 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:52,253 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:52,253 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:52,253 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:52,253 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:52,254 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:52,254 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:52,254 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:52,254 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:52,254 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:52,254 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,254 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:52,254 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:52,255 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,255 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:52,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:52,256 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:52,256 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:52,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:52,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:52,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d8fef7c[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:52,257 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:52,257 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:52,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:52,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567692254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,258 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:52,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:52,259 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:52,259 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,259 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:52,259 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:52,259 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:52,260 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,260 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:52,260 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:52,260 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:52,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:52,264 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:52,264 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:52,264 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:52,264 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:52,264 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:52,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:52,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567692262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,268 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,268 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:52,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,268 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:52,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:52,269 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:52,270 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:52,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:52,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,272 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:52,272 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:52,272 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:52,272 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:52,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:52,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:52,272 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:52,278 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:52,278 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,278 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:52,278 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:52,278 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,278 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:52,279 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:52,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:53:52,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:52,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:52,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567692279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:52,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567692279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,291 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/2d738780e9664a37a2a057454002a36b as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/2d738780e9664a37a2a057454002a36b 2023-04-27 03:53:52,299 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/B of 2b04f11bbacd80b8750d75eb2d0fbb35 into 2d738780e9664a37a2a057454002a36b(size=12.1 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:53:52,299 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:52,299 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/B, priority=12, startTime=1682567631818; duration=0sec 2023-04-27 03:53:52,299 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-27 03:53:52,299 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 4 compacting, 1 eligible, 16 blocking 2023-04-27 03:53:52,300 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-27 03:53:52,300 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-27 03:53:52,300 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. because compaction request was cancelled 2023-04-27 03:53:52,300 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 03:53:52,303 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60420 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 03:53:52,303 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/C is initiating minor compaction (all files) 2023-04-27 03:53:52,306 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/C in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:52,306 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/523c68810fea48a280fd571c0cfe79ce, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/1844e6156bce4dd995ea97da046f6bcd, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/9d8c985afa814970be62721548859585, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/3466f481f3844a8eb4eb68e4325e65f4, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/7470ae35249c4594867dfdba4ba1a4a5] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=59.0 K 2023-04-27 03:53:52,307 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 523c68810fea48a280fd571c0cfe79ce, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682567629515 2023-04-27 03:53:52,317 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 1844e6156bce4dd995ea97da046f6bcd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1682567629724 2023-04-27 03:53:52,318 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 9d8c985afa814970be62721548859585, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1682567630508 2023-04-27 03:53:52,319 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 3466f481f3844a8eb4eb68e4325e65f4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1682567631133 2023-04-27 03:53:52,320 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 7470ae35249c4594867dfdba4ba1a4a5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1682567631725 2023-04-27 03:53:52,321 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/1b0095364f4048eaa578abd6422275c8 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/1b0095364f4048eaa578abd6422275c8 2023-04-27 03:53:52,336 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/A of 2b04f11bbacd80b8750d75eb2d0fbb35 into 1b0095364f4048eaa578abd6422275c8(size=12.1 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:53:52,336 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:52,336 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/A, priority=12, startTime=1682567631818; duration=0sec 2023-04-27 03:53:52,336 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-27 03:53:52,336 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 5 compacting, 0 eligible, 16 blocking 2023-04-27 03:53:52,337 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-27 03:53:52,337 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-27 03:53:52,337 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. because compaction request was cancelled 2023-04-27 03:53:52,337 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-27 03:53:52,338 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-27 03:53:52,338 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-27 03:53:52,338 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. because compaction request was cancelled 2023-04-27 03:53:52,344 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#C#compaction#40 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:53:52,379 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:52,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:52,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:52,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:52,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:52,387 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:52,387 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:52,387 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:52,387 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:52,388 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,388 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:52,389 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:52,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:52,390 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,390 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:52,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:52,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:52,391 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:52,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:52,391 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:52,392 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:52,392 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:52,392 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:52,392 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:52,396 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:52,396 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:52,396 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:52,396 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:52,398 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:52,398 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:52,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:52,398 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:52,398 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:52,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:52,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:52,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:52,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:52,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:52,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12731379[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:52,401 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:52,401 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:52,404 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:52,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:52,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:52,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:52,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:52,405 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:52,405 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:52,406 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:52,408 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,408 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:52,408 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:52,408 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:52,409 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:52,409 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:52,409 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:52,409 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:52,410 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,410 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:52,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,410 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:52,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:52,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:52,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:52,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:52,413 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:52,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:52,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:52,415 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:52,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:52,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:52,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567692414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,420 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,420 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:52,420 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:52,420 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:52,420 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:52,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:52,420 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:52,421 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:52,421 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:52,421 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:52,421 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:52,421 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:52,421 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:52,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:52,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:52,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:52,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:53:52,480 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/491136f8d77f4563a052cf23a6393a8a 2023-04-27 03:53:52,505 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/90f3ed99ac784aa18ec3d95ed16a22ed 2023-04-27 03:53:52,514 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/2b064e1fef164267a9fe4bf5e76485c2 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/2b064e1fef164267a9fe4bf5e76485c2 2023-04-27 03:53:52,521 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:52,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:52,522 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/2b064e1fef164267a9fe4bf5e76485c2, entries=200, sequenceid=263, filesize=14.3 K 2023-04-27 03:53:52,525 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/491136f8d77f4563a052cf23a6393a8a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/491136f8d77f4563a052cf23a6393a8a 2023-04-27 03:53:52,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:52,542 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/491136f8d77f4563a052cf23a6393a8a, entries=150, sequenceid=263, filesize=11.9 K 2023-04-27 03:53:52,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:52,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:52,543 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:52,543 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:52,543 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:52,543 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:52,544 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/90f3ed99ac784aa18ec3d95ed16a22ed as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/90f3ed99ac784aa18ec3d95ed16a22ed 2023-04-27 03:53:52,544 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,544 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:52,546 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:52,546 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:52,546 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,546 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:52,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:52,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:52,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:52,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:52,557 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/90f3ed99ac784aa18ec3d95ed16a22ed, entries=150, sequenceid=263, filesize=11.9 K 2023-04-27 03:53:52,559 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~220.88 KB/226176, currentSize=100.63 KB/103050 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 570ms, sequenceid=263, compaction requested=true 2023-04-27 03:53:52,559 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:52,559 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:53:52,559 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:53:52,559 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:53:52,559 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-27 03:53:52,560 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:52,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:52,560 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:52,560 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:52,560 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:52,561 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41464 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:53:52,561 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/A is initiating minor compaction (all files) 2023-04-27 03:53:52,561 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/A in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:52,561 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/1b0095364f4048eaa578abd6422275c8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/92d8f0f012504a9ab9bd1b2778ee168e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/2b064e1fef164267a9fe4bf5e76485c2] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=40.5 K 2023-04-27 03:53:52,562 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 1b0095364f4048eaa578abd6422275c8, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1682567631133 2023-04-27 03:53:52,563 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 92d8f0f012504a9ab9bd1b2778ee168e, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1682567631721 2023-04-27 03:53:52,563 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 2b064e1fef164267a9fe4bf5e76485c2, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1682567631962 2023-04-27 03:53:52,565 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:52,574 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:52,574 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:52,574 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:52,574 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-27 03:53:52,574 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:53:52,574 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:52,574 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:53:52,575 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:52,575 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:53:52,575 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:52,607 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#A#compaction#43 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:53:52,640 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/95a843b25ff2431a84bff89c2dbf59be 2023-04-27 03:53:52,679 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/06d224ab17ac43a0b882f9588895bd86 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/06d224ab17ac43a0b882f9588895bd86 2023-04-27 03:53:52,687 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/A of 2b04f11bbacd80b8750d75eb2d0fbb35 into 06d224ab17ac43a0b882f9588895bd86(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:53:52,688 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:52,688 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/A, priority=13, startTime=1682567632559; duration=0sec 2023-04-27 03:53:52,688 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-27 03:53:52,688 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 5 compacting, 1 eligible, 16 blocking 2023-04-27 03:53:52,689 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-27 03:53:52,689 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-27 03:53:52,689 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. because compaction request was cancelled 2023-04-27 03:53:52,689 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:53:52,691 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36634 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:53:52,691 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/B is initiating minor compaction (all files) 2023-04-27 03:53:52,692 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/B in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:52,692 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/2d738780e9664a37a2a057454002a36b, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/3d67f32ca8f94546abad605f17dc4f77, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/491136f8d77f4563a052cf23a6393a8a] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=35.8 K 2023-04-27 03:53:52,692 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/45d82f3b8baf4b5c98678f7beb633e97 2023-04-27 03:53:52,692 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 2d738780e9664a37a2a057454002a36b, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1682567631133 2023-04-27 03:53:52,696 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 3d67f32ca8f94546abad605f17dc4f77, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1682567631725 2023-04-27 03:53:52,697 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 491136f8d77f4563a052cf23a6393a8a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1682567631976 2023-04-27 03:53:52,725 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#B#compaction#46 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:53:52,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:52,725 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:52,770 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/d07450635f734fffb96287257b67985b as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/d07450635f734fffb96287257b67985b 2023-04-27 03:53:52,772 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/ad5abd13e84e498082c7f169a7bb07d7 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/ad5abd13e84e498082c7f169a7bb07d7 2023-04-27 03:53:52,785 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/B of 2b04f11bbacd80b8750d75eb2d0fbb35 into ad5abd13e84e498082c7f169a7bb07d7(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:53:52,785 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:52,786 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/B, priority=13, startTime=1682567632559; duration=0sec 2023-04-27 03:53:52,786 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:53:52,786 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/C of 2b04f11bbacd80b8750d75eb2d0fbb35 into d07450635f734fffb96287257b67985b(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:53:52,787 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:52,787 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/C, priority=11, startTime=1682567631988; duration=0sec 2023-04-27 03:53:52,787 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:53:52,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:52,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567692792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:52,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:52,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567692904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:53,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567693107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,138 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/eb37be95781349ceaf05d20f61f4b2eb 2023-04-27 03:53:53,146 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/95a843b25ff2431a84bff89c2dbf59be as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/95a843b25ff2431a84bff89c2dbf59be 2023-04-27 03:53:53,152 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/95a843b25ff2431a84bff89c2dbf59be, entries=150, sequenceid=283, filesize=11.9 K 2023-04-27 03:53:53,153 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/45d82f3b8baf4b5c98678f7beb633e97 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/45d82f3b8baf4b5c98678f7beb633e97 2023-04-27 03:53:53,159 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/45d82f3b8baf4b5c98678f7beb633e97, entries=150, sequenceid=283, filesize=11.9 K 2023-04-27 03:53:53,160 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/eb37be95781349ceaf05d20f61f4b2eb as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/eb37be95781349ceaf05d20f61f4b2eb 2023-04-27 03:53:53,170 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/eb37be95781349ceaf05d20f61f4b2eb, entries=150, sequenceid=283, filesize=11.9 K 2023-04-27 03:53:53,171 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 597ms, sequenceid=283, compaction requested=true 2023-04-27 03:53:53,171 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:53,171 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:53,171 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:53,173 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:53,173 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,173 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:53,173 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:53,174 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,174 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:53,178 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:53,178 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:53,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:53,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22a90bcd[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:53,180 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:53,180 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:53,182 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:53,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:53,182 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:53,182 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:53,182 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:53,182 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,182 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:53,183 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,183 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:53,184 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:53,184 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:53,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:53,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:53,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:53,187 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:53,187 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:53,187 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:53,187 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:53,187 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:53,210 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:53,210 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:53,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:53,210 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:53,211 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:53,212 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:53,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,233 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:53,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:53:53,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:53,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:53,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:53,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:53,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:53,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,234 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:53,234 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:53,234 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:53,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:53,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:53,234 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:53,234 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:53,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:53,263 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-27 03:53:53,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:53:53,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:53,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:53:53,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:53,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:53:53,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:53,284 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/152c07b5916844fc9233f68601fc337e 2023-04-27 03:53:53,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:53,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567693321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:53,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:53,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567693321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567693322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:53,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567693322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,334 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:53,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:53,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:53,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:53,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:53,339 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:53,339 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:53,339 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:53,339 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:53,340 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:53,340 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:53,387 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,387 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:53,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:53,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:53,387 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,387 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:53,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:53,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:53,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:53,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:53,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:53,389 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:53,389 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:53,389 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:53,389 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:53,389 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:53,392 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:53,392 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:53,392 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:53,392 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:53,392 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:53,392 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:53,392 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:53,392 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:53,392 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,392 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:53,392 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:53,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:53,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567693416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:53,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567693429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:53,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567693431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:53,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567693432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:53,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567693433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,471 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:53,471 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:53,473 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:53,473 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:53,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:53,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18cb5373[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:53,476 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:53,476 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:53,541 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:53,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:53,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:53,541 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:53,541 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:53,541 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:53,542 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,542 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:53,594 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:53,594 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:53,595 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:53,595 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:53,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:53,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:53,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:53,601 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:53,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:53,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:53,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:53,601 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:53,625 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:53,625 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:53,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:53,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:53,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:53,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:53,625 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:53,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:53,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:53,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:53,627 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:53,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:53,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:53,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,629 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:53,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:53,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:53,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567693635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567693635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:53,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567693640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:53,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567693646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,671 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:53,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:53,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:53,672 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:53,672 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,672 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:53,672 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:53,672 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,672 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:53,672 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:53,672 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:53,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:53,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:53,672 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:53,673 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:53,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:53,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:53,673 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:53,709 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/5c8817e801a347b580d67b2ef502e8cd 2023-04-27 03:53:53,725 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/6e476e4656bf4d9697923e4f56073113 2023-04-27 03:53:53,732 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/152c07b5916844fc9233f68601fc337e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/152c07b5916844fc9233f68601fc337e 2023-04-27 03:53:53,740 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/152c07b5916844fc9233f68601fc337e, entries=200, sequenceid=301, filesize=14.3 K 2023-04-27 03:53:53,741 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/5c8817e801a347b580d67b2ef502e8cd as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/5c8817e801a347b580d67b2ef502e8cd 2023-04-27 03:53:53,748 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/5c8817e801a347b580d67b2ef502e8cd, entries=150, sequenceid=301, filesize=11.9 K 2023-04-27 03:53:53,749 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/6e476e4656bf4d9697923e4f56073113 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/6e476e4656bf4d9697923e4f56073113 2023-04-27 03:53:53,761 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/6e476e4656bf4d9697923e4f56073113, entries=150, sequenceid=301, filesize=11.9 K 2023-04-27 03:53:53,763 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 499ms, sequenceid=301, compaction requested=true 2023-04-27 03:53:53,763 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:53,763 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:53:53,763 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:53:53,763 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:53:53,763 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:53:53,763 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:53:53,765 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39476 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:53:53,765 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/A is initiating minor compaction (all files) 2023-04-27 03:53:53,765 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/A in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:53,765 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/06d224ab17ac43a0b882f9588895bd86, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/95a843b25ff2431a84bff89c2dbf59be, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/152c07b5916844fc9233f68601fc337e] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=38.6 K 2023-04-27 03:53:53,766 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 06d224ab17ac43a0b882f9588895bd86, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1682567631976 2023-04-27 03:53:53,767 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:53:53,768 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/B is initiating minor compaction (all files) 2023-04-27 03:53:53,768 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/B in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:53,768 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 95a843b25ff2431a84bff89c2dbf59be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1682567632037 2023-04-27 03:53:53,768 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/ad5abd13e84e498082c7f169a7bb07d7, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/45d82f3b8baf4b5c98678f7beb633e97, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/5c8817e801a347b580d67b2ef502e8cd] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=36.2 K 2023-04-27 03:53:53,769 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 152c07b5916844fc9233f68601fc337e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1682567632743 2023-04-27 03:53:53,769 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting ad5abd13e84e498082c7f169a7bb07d7, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1682567631976 2023-04-27 03:53:53,771 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 45d82f3b8baf4b5c98678f7beb633e97, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1682567632037 2023-04-27 03:53:53,772 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 5c8817e801a347b580d67b2ef502e8cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1682567632743 2023-04-27 03:53:53,773 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:53,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:53,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:53,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:53,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:53,780 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:53,780 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:53,780 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:53,780 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:53,781 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:53,781 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:53,793 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#B#compaction#50 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:53:53,795 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#A#compaction#51 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:53:53,810 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,810 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:53,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:53,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:53,810 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,810 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:53,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:53,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:53,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:53,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:53,816 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:53,816 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:53,816 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:53,820 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:53,820 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:53,820 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:53,820 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:53,820 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-27 03:53:53,821 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:53:53,821 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:53,821 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:53:53,821 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:53,821 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:53:53,821 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:53,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:53,934 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:53,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:53,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567693957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:53,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567693958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:53,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567693960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:53,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567693961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:53,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:53,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567693964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567694065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567694066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567694070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567694070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567694071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,234 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/a7f44a67a6ed45e5835a6f5573a351b8 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/a7f44a67a6ed45e5835a6f5573a351b8 2023-04-27 03:53:54,234 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/cfde396119634d3ca179bb1abca8db21 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/cfde396119634d3ca179bb1abca8db21 2023-04-27 03:53:54,242 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/9e9266c4a08147b7834a3a0910518bb8 2023-04-27 03:53:54,254 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/B of 2b04f11bbacd80b8750d75eb2d0fbb35 into a7f44a67a6ed45e5835a6f5573a351b8(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:53:54,254 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:54,254 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/B, priority=13, startTime=1682567633763; duration=0sec 2023-04-27 03:53:54,254 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:53:54,255 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/A of 2b04f11bbacd80b8750d75eb2d0fbb35 into cfde396119634d3ca179bb1abca8db21(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:53:54,255 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:54,255 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/A, priority=13, startTime=1682567633763; duration=0sec 2023-04-27 03:53:54,255 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 03:53:54,255 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:53:54,259 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49004 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 03:53:54,259 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/C is initiating minor compaction (all files) 2023-04-27 03:53:54,260 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/C in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:54,260 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/d07450635f734fffb96287257b67985b, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/90f3ed99ac784aa18ec3d95ed16a22ed, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/eb37be95781349ceaf05d20f61f4b2eb, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/6e476e4656bf4d9697923e4f56073113] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=47.9 K 2023-04-27 03:53:54,262 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting d07450635f734fffb96287257b67985b, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1682567631725 2023-04-27 03:53:54,263 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 90f3ed99ac784aa18ec3d95ed16a22ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1682567631976 2023-04-27 03:53:54,263 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting eb37be95781349ceaf05d20f61f4b2eb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1682567632037 2023-04-27 03:53:54,264 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 6e476e4656bf4d9697923e4f56073113, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1682567632743 2023-04-27 03:53:54,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567694271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567694272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567694277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567694278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567694278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,291 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#C#compaction#54 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:53:54,293 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/ace675ba25b840d3bce9d5a524be25fe 2023-04-27 03:53:54,318 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/8a9b47b7bfdf458594951f04aef3029d 2023-04-27 03:53:54,324 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/9e9266c4a08147b7834a3a0910518bb8 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/9e9266c4a08147b7834a3a0910518bb8 2023-04-27 03:53:54,330 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/9e9266c4a08147b7834a3a0910518bb8, entries=150, sequenceid=318, filesize=11.9 K 2023-04-27 03:53:54,331 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/ace675ba25b840d3bce9d5a524be25fe as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/ace675ba25b840d3bce9d5a524be25fe 2023-04-27 03:53:54,338 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/ace675ba25b840d3bce9d5a524be25fe, entries=150, sequenceid=318, filesize=11.9 K 2023-04-27 03:53:54,339 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/8a9b47b7bfdf458594951f04aef3029d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/8a9b47b7bfdf458594951f04aef3029d 2023-04-27 03:53:54,346 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/8a9b47b7bfdf458594951f04aef3029d, entries=150, sequenceid=318, filesize=11.9 K 2023-04-27 03:53:54,347 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 527ms, sequenceid=318, compaction requested=false 2023-04-27 03:53:54,347 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:54,347 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:54,347 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:54,348 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:54,348 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,348 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:54,348 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:54,350 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,350 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:54,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:54,350 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,350 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:54,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:54,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:54,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:54,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:54,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:54,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a6dab19[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:54,353 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:54,353 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,354 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,355 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:54,355 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:54,355 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,355 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:54,355 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:54,356 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,356 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:54,357 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:54,357 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:54,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:54,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:54,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:54,372 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:54,372 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:54,372 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:54,372 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:54,372 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:54,375 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,375 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,375 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:54,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:54,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,376 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:54,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,378 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:54,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:54,388 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:54,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:54,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,389 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:54,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:54,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:54,389 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:54,389 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:54,390 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:54,390 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:54,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:54,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:54,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:54,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:54,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:53:54,489 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:54,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:54,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:54,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:54,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:54,495 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:54,495 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:54,495 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:54,495 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:54,495 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,495 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:54,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:54,497 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:54,497 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,497 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:54,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:54,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:54,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:54,500 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:54,500 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:54,500 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:54,501 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:54,501 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:54,501 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:54,501 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:54,501 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-27 03:53:54,501 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:53:54,501 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:54,501 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:53:54,501 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:54,501 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:53:54,502 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:54,518 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/f56179240e5749f29f78628dc5fedff4 2023-04-27 03:53:54,546 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/e5317f30301b46349a7c3f2d21857919 2023-04-27 03:53:54,567 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/7facb8e0cf284125af54a07b20af81e4 2023-04-27 03:53:54,578 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/f56179240e5749f29f78628dc5fedff4 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/f56179240e5749f29f78628dc5fedff4 2023-04-27 03:53:54,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:54,582 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:54,587 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/f56179240e5749f29f78628dc5fedff4, entries=150, sequenceid=335, filesize=11.9 K 2023-04-27 03:53:54,591 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/e5317f30301b46349a7c3f2d21857919 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/e5317f30301b46349a7c3f2d21857919 2023-04-27 03:53:54,597 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/e5317f30301b46349a7c3f2d21857919, entries=150, sequenceid=335, filesize=11.9 K 2023-04-27 03:53:54,599 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/7facb8e0cf284125af54a07b20af81e4 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/7facb8e0cf284125af54a07b20af81e4 2023-04-27 03:53:54,604 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/7facb8e0cf284125af54a07b20af81e4, entries=150, sequenceid=335, filesize=11.9 K 2023-04-27 03:53:54,606 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=87.22 KB/89310 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 105ms, sequenceid=335, compaction requested=true 2023-04-27 03:53:54,606 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:54,606 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:54,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:54,609 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:54,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:54,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:54,611 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,611 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:54,612 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,612 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:54,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:54,613 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-27 03:53:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:54,613 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:53:54,613 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:54,613 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:53:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,613 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:54,613 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:53:54,613 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:54,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:54,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@408f91f6[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:54,614 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:54,614 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,616 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,616 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:54,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,617 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:54,617 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:54,617 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:54,620 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:54,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:54,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:54,624 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:54,624 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:54,624 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:54,624 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:54,624 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:54,626 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,626 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:54,626 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:54,628 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:54,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567694624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:54,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567694625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,630 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:54,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567694626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,631 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567694628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,631 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567694629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,634 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,634 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:54,634 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:54,634 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,634 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,634 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:54,634 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:54,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:54,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:54,634 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:54,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:54,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:54,635 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:54,635 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:54,634 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:54,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:54,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:53:54,635 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,638 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/8080140782a044cd880cac811a541f35 2023-04-27 03:53:54,722 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/6aefb67646fb488ca273132a6e53d174 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/6aefb67646fb488ca273132a6e53d174 2023-04-27 03:53:54,731 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/C of 2b04f11bbacd80b8750d75eb2d0fbb35 into 6aefb67646fb488ca273132a6e53d174(size=12.4 K), total size for store is 36.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:53:54,732 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:54,732 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/C, priority=12, startTime=1682567633763; duration=0sec 2023-04-27 03:53:54,732 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:53:54,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567694731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567694731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,736 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:54,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567694732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567694733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567694733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:54,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:54,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:54,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:54,744 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:54,744 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:54,744 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:54,744 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:54,745 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,745 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:54,748 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:54,748 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:54,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:54,748 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,748 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:54,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:54,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:54,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:54,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:54,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:54,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:54,750 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:54,750 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:54,750 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:54,752 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:54,752 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:54,752 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:54,752 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:54,752 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:54,752 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:54,752 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:54,752 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:54,752 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,752 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:54,753 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:54,754 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,754 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:54,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:54,758 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,758 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:54,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:54,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a120d49[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:54,761 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:54,761 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,762 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,762 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:54,762 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:54,762 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:54,762 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,762 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:54,764 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,764 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:54,764 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:54,764 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:54,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:54,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:54,770 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:54,770 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:54,770 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:54,770 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:54,770 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:54,774 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,774 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:54,774 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:54,775 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:54,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:54,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:54,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,777 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,780 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:54,780 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,780 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:54,780 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:54,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:54,780 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:54,781 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:54,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:54,780 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:54,781 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,781 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,781 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:54,781 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:54,781 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:54,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:54,881 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:54,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:54,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:54,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:54,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:54,891 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:54,891 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:54,891 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:54,891 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:54,892 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,892 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:54,893 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,894 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,894 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:54,894 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:54,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:54,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:54,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:54,899 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:54,909 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:54,909 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:54,909 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:54,909 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:54,909 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:54,912 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:54,912 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:54,912 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:54,912 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:54,912 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:54,916 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,916 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:54,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:54,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:54,916 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,917 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:54,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:54,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:54,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:54,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@680093b3[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:54,918 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:54,918 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,921 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,921 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:54,921 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:54,921 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:54,921 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,921 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:54,923 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:54,923 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:54,923 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:54,924 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:54,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:54,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:54,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:54,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:54,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:54,927 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:54,927 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:54,927 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:54,927 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:54,927 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:54,929 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,929 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:54,929 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:54,930 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:54,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:54,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,932 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:54,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,936 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:54,936 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:54,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:54,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:54,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:54,936 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:54,936 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,937 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:54,937 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:54,937 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,937 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:54,937 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,937 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:54,937 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:54,937 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:54,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:54,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:54,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567694936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567694939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567694939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567694939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:54,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:54,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567694947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,037 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:55,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:55,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:55,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:55,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:55,042 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:55,042 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:55,043 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:55,043 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:55,043 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,043 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:55,045 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:55,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:55,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:55,047 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:55,048 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,048 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:55,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:55,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:55,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:55,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:55,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:55,050 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:55,050 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:55,050 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:55,051 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:55,051 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:55,051 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:55,051 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:55,051 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:55,051 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:55,051 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:55,051 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:55,051 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,051 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:55,051 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:55,056 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:55,057 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:55,058 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,058 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:55,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:55,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:55,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f12c8c9[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:55,059 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:55,059 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,060 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,060 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,061 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:55,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,061 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/f9b0a1f63a5b402a981a2436aa0efcff 2023-04-27 03:53:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,061 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:55,061 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:55,061 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:55,063 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:55,063 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:55,063 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:55,064 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:55,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:55,067 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:55,067 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:55,067 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:55,067 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:55,067 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:55,069 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,069 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:55,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,069 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,071 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:55,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:55,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:55,077 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:55,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,085 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:55,085 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:55,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:55,085 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,085 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:55,085 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:55,085 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,086 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,086 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,086 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:55,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:55,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:55,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:55,086 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:55,086 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:55,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:53:55,085 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:55,186 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:55,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:55,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:55,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:55,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:55,197 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:55,197 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:55,197 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:55,197 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:55,198 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,199 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:55,213 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:55,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:55,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:55,213 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:55,214 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,214 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:55,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:55,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:55,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:55,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:55,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:55,219 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:55,219 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:55,219 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:55,224 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:55,224 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:55,225 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:55,225 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:55,225 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:55,225 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:55,225 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:55,225 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:55,225 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,225 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:55,225 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:55,227 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,227 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:55,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:55,227 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,227 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:55,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:55,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:55,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:55,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:55,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@732ee0fc[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:55,231 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:55,231 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:55,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:55,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:55,234 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,235 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:55,237 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,237 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:55,237 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:55,237 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:55,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:55,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:55,242 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:55,242 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:55,242 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:55,242 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:55,242 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:55,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:55,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567695243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:55,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567695243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,246 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:55,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:55,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567695244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,246 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:55,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:55,246 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:55,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567695244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:55,247 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,249 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,251 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,252 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:55,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:55,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:55,252 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:55,252 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:55,252 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,252 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,252 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,252 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:55,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:53:55,252 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:55,252 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:55,252 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:55,252 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:55,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:55,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:55,254 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:55,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567695252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,352 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:55,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:55,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:55,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:55,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:55,360 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:55,360 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:55,360 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:55,360 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:55,361 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,361 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:55,363 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:55,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:55,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:55,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:55,364 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:55,366 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:55,366 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,366 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:55,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:55,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:55,372 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:55,372 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:55,372 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:55,380 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:55,380 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:55,380 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:55,380 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:55,381 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:55,381 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:55,381 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:55,381 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:55,381 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,381 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:55,381 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:55,384 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:55,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:55,384 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:55,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:55,385 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,386 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:55,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:55,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:55,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:55,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49ecd395[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:55,388 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:55,388 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,390 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:55,390 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:55,390 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:55,390 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,390 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:55,391 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,391 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:55,392 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:55,392 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:55,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:55,397 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:55,397 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:55,397 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:55,397 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:55,397 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:55,400 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,400 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:55,400 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:55,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,402 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:55,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:55,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,404 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,409 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:55,409 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:55,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:55,409 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,410 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:55,410 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:55,410 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:55,410 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:55,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:55,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:55,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:53:55,410 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:55,410 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:55,410 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,411 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,411 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:55,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:55,510 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:55,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:55,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:55,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:55,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:55,519 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/8714328911c64eb58e76e84361a83caf 2023-04-27 03:53:55,519 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:55,519 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:55,519 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:55,519 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:55,520 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,520 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:55,522 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:55,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:55,522 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:55,523 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,523 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:55,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:55,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:55,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:55,525 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:55,525 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:55,525 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:55,525 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:55,527 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/8080140782a044cd880cac811a541f35 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/8080140782a044cd880cac811a541f35 2023-04-27 03:53:55,528 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:55,528 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:55,528 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:55,528 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:55,528 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:55,529 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:55,529 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:55,529 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:55,529 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,529 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:55,529 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:55,532 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,532 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:55,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:55,533 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,533 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:55,535 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/8080140782a044cd880cac811a541f35, entries=150, sequenceid=353, filesize=11.9 K 2023-04-27 03:53:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:55,536 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/f9b0a1f63a5b402a981a2436aa0efcff as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/f9b0a1f63a5b402a981a2436aa0efcff 2023-04-27 03:53:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:55,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:55,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@251d5c10[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:55,538 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:55,538 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,543 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/f9b0a1f63a5b402a981a2436aa0efcff, entries=150, sequenceid=353, filesize=11.9 K 2023-04-27 03:53:55,545 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/8714328911c64eb58e76e84361a83caf as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/8714328911c64eb58e76e84361a83caf 2023-04-27 03:53:55,548 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,548 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,548 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:55,548 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:55,548 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:55,548 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:55,550 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:55,550 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,550 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:55,551 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:55,554 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/8714328911c64eb58e76e84361a83caf, entries=150, sequenceid=353, filesize=11.9 K 2023-04-27 03:53:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,556 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 943ms, sequenceid=353, compaction requested=true 2023-04-27 03:53:55,556 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:55,556 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 03:53:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:55,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:55,556 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:55,556 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:53:55,556 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:55,557 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:55,557 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 03:53:55,557 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:55,557 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:55,557 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:53:55,557 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:53:55,558 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 03:53:55,559 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/A is initiating minor compaction (all files) 2023-04-27 03:53:55,559 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 03:53:55,559 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/A in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:55,559 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/B is initiating minor compaction (all files) 2023-04-27 03:53:55,559 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/B in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:55,559 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/cfde396119634d3ca179bb1abca8db21, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/9e9266c4a08147b7834a3a0910518bb8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/f56179240e5749f29f78628dc5fedff4, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/8080140782a044cd880cac811a541f35] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=48.2 K 2023-04-27 03:53:55,559 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/a7f44a67a6ed45e5835a6f5573a351b8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/ace675ba25b840d3bce9d5a524be25fe, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/e5317f30301b46349a7c3f2d21857919, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/f9b0a1f63a5b402a981a2436aa0efcff] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=48.2 K 2023-04-27 03:53:55,560 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting cfde396119634d3ca179bb1abca8db21, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1682567632743 2023-04-27 03:53:55,560 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting a7f44a67a6ed45e5835a6f5573a351b8, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1682567632743 2023-04-27 03:53:55,560 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 9e9266c4a08147b7834a3a0910518bb8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1682567633316 2023-04-27 03:53:55,561 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting ace675ba25b840d3bce9d5a524be25fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1682567633316 2023-04-27 03:53:55,561 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting f56179240e5749f29f78628dc5fedff4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1682567633942 2023-04-27 03:53:55,561 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting e5317f30301b46349a7c3f2d21857919, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1682567633942 2023-04-27 03:53:55,562 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 8080140782a044cd880cac811a541f35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1682567634598 2023-04-27 03:53:55,562 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting f9b0a1f63a5b402a981a2436aa0efcff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1682567634598 2023-04-27 03:53:55,572 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,572 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:55,572 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:55,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:55,573 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:55,577 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#A#compaction#62 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:53:55,579 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,580 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#B#compaction#63 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:53:55,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:55,581 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,581 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:55,581 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:55,581 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:55,582 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:55,582 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:55,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:55,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:55,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:55,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:55,582 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,582 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,582 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:55,582 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:55,582 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:55,582 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:55,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:55,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:53:55,603 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/7f8bafed60a14718bd366d4b528b4894 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/7f8bafed60a14718bd366d4b528b4894 2023-04-27 03:53:55,613 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/B of 2b04f11bbacd80b8750d75eb2d0fbb35 into 7f8bafed60a14718bd366d4b528b4894(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:53:55,613 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:55,613 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/B, priority=12, startTime=1682567635556; duration=0sec 2023-04-27 03:53:55,614 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:53:55,614 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 03:53:55,618 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49290 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 03:53:55,619 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/C is initiating minor compaction (all files) 2023-04-27 03:53:55,619 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/C in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:55,619 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/6aefb67646fb488ca273132a6e53d174, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/8a9b47b7bfdf458594951f04aef3029d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/7facb8e0cf284125af54a07b20af81e4, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/8714328911c64eb58e76e84361a83caf] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=48.1 K 2023-04-27 03:53:55,619 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 6aefb67646fb488ca273132a6e53d174, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1682567632743 2023-04-27 03:53:55,620 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 8a9b47b7bfdf458594951f04aef3029d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1682567633316 2023-04-27 03:53:55,623 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 7facb8e0cf284125af54a07b20af81e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1682567633942 2023-04-27 03:53:55,624 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 8714328911c64eb58e76e84361a83caf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1682567634598 2023-04-27 03:53:55,626 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/b4312b8e1f904a1b9ff7cfe501e5d976 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/b4312b8e1f904a1b9ff7cfe501e5d976 2023-04-27 03:53:55,636 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/A of 2b04f11bbacd80b8750d75eb2d0fbb35 into b4312b8e1f904a1b9ff7cfe501e5d976(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:53:55,636 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:55,636 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/A, priority=12, startTime=1682567635556; duration=0sec 2023-04-27 03:53:55,637 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:53:55,650 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#C#compaction#64 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:53:55,682 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:55,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:55,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:55,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:55,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:55,700 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:55,700 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:55,700 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:55,700 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:55,702 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,702 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:55,704 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:55,704 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:55,705 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,705 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:55,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:55,707 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:55,716 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:55,716 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:55,716 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:55,716 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:55,716 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:55,716 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:55,717 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:55,717 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:55,717 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:55,717 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-27 03:53:55,719 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:53:55,720 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:55,720 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:53:55,720 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:55,724 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:53:55,724 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:55,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:55,751 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:55,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:55,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:55,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567695777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567695776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:55,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567695778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:55,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567695780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:55,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567695780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:55,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567695881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:55,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567695883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:55,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567695883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:55,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567695883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:55,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:55,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567695883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:56,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:56,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50850 deadline: 1682567696085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567696086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:56,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567696086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:56,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567696090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:56,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567696090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,099 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/81c2e0edd58b493cb31a31e653615a2f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/81c2e0edd58b493cb31a31e653615a2f 2023-04-27 03:53:56,106 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/C of 2b04f11bbacd80b8750d75eb2d0fbb35 into 81c2e0edd58b493cb31a31e653615a2f(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:53:56,106 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:56,106 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/C, priority=12, startTime=1682567635557; duration=0sec 2023-04-27 03:53:56,107 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:53:56,143 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/112c6f6dab5c438892f42025c696582e 2023-04-27 03:53:56,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:56,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50862 deadline: 1682567696389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:56,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567696390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:56,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567696391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,394 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:56,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567696393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:56,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50852 deadline: 1682567696397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,562 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/06c006c3168d44b485137cf9e63579d8 2023-04-27 03:53:56,596 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/b2940039c8e645b2b3686905d2c7b596 2023-04-27 03:53:56,613 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/112c6f6dab5c438892f42025c696582e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/112c6f6dab5c438892f42025c696582e 2023-04-27 03:53:56,620 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/112c6f6dab5c438892f42025c696582e, entries=150, sequenceid=370, filesize=11.9 K 2023-04-27 03:53:56,621 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/06c006c3168d44b485137cf9e63579d8 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/06c006c3168d44b485137cf9e63579d8 2023-04-27 03:53:56,627 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/06c006c3168d44b485137cf9e63579d8, entries=150, sequenceid=370, filesize=11.9 K 2023-04-27 03:53:56,629 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/b2940039c8e645b2b3686905d2c7b596 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/b2940039c8e645b2b3686905d2c7b596 2023-04-27 03:53:56,636 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/b2940039c8e645b2b3686905d2c7b596, entries=150, sequenceid=370, filesize=11.9 K 2023-04-27 03:53:56,637 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 920ms, sequenceid=370, compaction requested=false 2023-04-27 03:53:56,637 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:56,637 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:56,637 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:56,637 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:56,637 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,637 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:56,637 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:56,639 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:56,639 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:56,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:56,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:56,639 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:56,640 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:56,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:56,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:56,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:56,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:56,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c909cfa[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:56,642 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:56,642 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:56,645 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:56,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:56,645 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:56,645 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:56,645 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:56,645 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,645 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:56,647 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:56,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:56,647 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:56,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:56,648 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:56,648 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:56,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:56,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:56,652 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:56,652 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:56,652 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:56,652 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:56,652 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:56,655 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:56,655 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:56,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:56,655 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:56,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:56,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:56,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:56,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:56,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:56,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:56,656 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:56,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,658 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:56,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:56,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,661 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:56,661 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:56,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:56,661 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,661 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:56,661 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:56,661 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:56,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:56,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:56,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:53:56,662 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:56,662 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:56,661 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,662 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:56,662 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:56,662 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:56,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:56,762 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:56,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:56,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:56,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:56,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:56,766 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:56,766 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:56,766 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:56,766 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:56,767 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:56,767 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:56,769 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:56,769 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:56,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:56,770 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,770 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:56,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:56,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:56,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:56,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:56,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:56,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:56,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:56,771 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:56,771 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:56,771 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:56,776 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:56,776 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:56,776 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:56,776 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:56,776 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-27 03:53:56,776 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:53:56,777 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:56,777 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:53:56,777 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:56,777 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:53:56,777 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:56,794 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/543aa6d428d043e1863d36d2e71cb9a1 2023-04-27 03:53:56,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:56,894 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:56,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:56,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567696902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:56,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567696903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:56,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567696904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:56,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567696905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:56,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:56,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50928 deadline: 1682567696906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567697006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567697006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567697007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,011 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567697008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567697011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,210 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/628e24b6f9e24bf8bd3d4cfaa28c4d8a 2023-04-27 03:53:57,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567697211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567697211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567697211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567697214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567697214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,231 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/6be0734d02c54fe4aa7a1ec89bed28de 2023-04-27 03:53:57,237 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/543aa6d428d043e1863d36d2e71cb9a1 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/543aa6d428d043e1863d36d2e71cb9a1 2023-04-27 03:53:57,242 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/543aa6d428d043e1863d36d2e71cb9a1, entries=150, sequenceid=390, filesize=11.9 K 2023-04-27 03:53:57,243 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/628e24b6f9e24bf8bd3d4cfaa28c4d8a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/628e24b6f9e24bf8bd3d4cfaa28c4d8a 2023-04-27 03:53:57,248 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/628e24b6f9e24bf8bd3d4cfaa28c4d8a, entries=150, sequenceid=390, filesize=11.9 K 2023-04-27 03:53:57,249 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/6be0734d02c54fe4aa7a1ec89bed28de as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/6be0734d02c54fe4aa7a1ec89bed28de 2023-04-27 03:53:57,266 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/6be0734d02c54fe4aa7a1ec89bed28de, entries=150, sequenceid=390, filesize=11.9 K 2023-04-27 03:53:57,267 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 491ms, sequenceid=390, compaction requested=true 2023-04-27 03:53:57,267 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:57,267 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:57,267 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:57,268 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:57,268 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,268 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:57,268 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:57,269 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,269 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:57,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:57,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:57,270 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:57,270 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:57,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:57,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:57,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:57,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:57,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:57,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78bdba94[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:57,272 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:57,272 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:57,273 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:57,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:57,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:57,273 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,273 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:57,273 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:57,273 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:57,273 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:57,275 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,275 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:57,275 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:57,275 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:57,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:57,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:57,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:57,278 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:57,278 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:57,278 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:57,278 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:57,278 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:57,281 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:57,281 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:57,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:57,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:57,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:57,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:57,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:57,281 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:57,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:57,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:57,282 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:57,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:57,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,283 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:57,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:57,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,286 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:57,286 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,286 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:57,286 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:57,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:57,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:57,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:57,287 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:57,287 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:57,286 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:57,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:57,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:53:57,286 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:57,287 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,287 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:57,287 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:57,287 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:57,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:57,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:57,387 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:57,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:57,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:57,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:57,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:57,392 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:57,392 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:57,392 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:57,392 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:57,393 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:57,393 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:57,394 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:57,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:57,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:57,394 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:57,395 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,395 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:57,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:57,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:57,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:57,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:57,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:57,397 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:57,397 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:57,397 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:57,400 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:57,400 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:57,400 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:57,400 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:57,400 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-27 03:53:57,400 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:53:57,400 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:57,400 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:53:57,400 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:57,400 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:53:57,400 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:57,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:57,515 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:57,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567697530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,535 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567697533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567697533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567697534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567697534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567697635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,639 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567697637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567697637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567697637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567697637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,811 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/1929a49af4d0450183b1a5907a45c23a 2023-04-27 03:53:57,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567697838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567697840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567697845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567697846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:57,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:57,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567697846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:58,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:58,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567698143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567698143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:58,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567698150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:58,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567698154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:58,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567698155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,235 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/f7296eef8d0c4e3095044897ad3b9baa 2023-04-27 03:53:58,649 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/79b4152c94d54c0eb69d65f6d01600f6 2023-04-27 03:53:58,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:58,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567698651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:58,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567698651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:58,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567698655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,659 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/1929a49af4d0450183b1a5907a45c23a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/1929a49af4d0450183b1a5907a45c23a 2023-04-27 03:53:58,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:58,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567698658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:58,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567698660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,665 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/1929a49af4d0450183b1a5907a45c23a, entries=150, sequenceid=404, filesize=11.9 K 2023-04-27 03:53:58,666 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/f7296eef8d0c4e3095044897ad3b9baa as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/f7296eef8d0c4e3095044897ad3b9baa 2023-04-27 03:53:58,671 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/f7296eef8d0c4e3095044897ad3b9baa, entries=150, sequenceid=404, filesize=11.9 K 2023-04-27 03:53:58,672 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/79b4152c94d54c0eb69d65f6d01600f6 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/79b4152c94d54c0eb69d65f6d01600f6 2023-04-27 03:53:58,678 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/79b4152c94d54c0eb69d65f6d01600f6, entries=150, sequenceid=404, filesize=11.9 K 2023-04-27 03:53:58,679 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 1279ms, sequenceid=404, compaction requested=true 2023-04-27 03:53:58,679 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:58,679 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:58,679 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:58,679 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:58,679 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,679 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:58,679 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:58,681 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,681 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:58,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:58,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:58,681 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:58,681 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:58,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:58,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:58,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:58,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:58,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:58,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5762be6d[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:58,683 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:58,683 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:58,684 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:58,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:58,684 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:58,684 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:58,684 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:58,684 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,684 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:58,690 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,690 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:58,690 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:58,690 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:58,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:58,693 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:58,693 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:58,694 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:58,694 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:58,694 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:58,695 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:58,695 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:58,695 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:58,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:58,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:58,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:58,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:58,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:58,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:58,696 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:58,697 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,700 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,700 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:58,700 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:58,700 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:58,700 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:58,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:58,700 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:58,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:53:58,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:58,700 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:58,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:58,700 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:58,700 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:58,700 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:58,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:58,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:58,700 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:58,701 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:58,801 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:53:58,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:53:58,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:53:58,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:53:58,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:53:58,824 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:53:58,824 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:58,824 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:53:58,824 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:53:58,825 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:58,825 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:58,826 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,826 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:58,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:58,827 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:58,827 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:53:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:58,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:53:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:58,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:53:58,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:53:58,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:53:58,832 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:53:58,832 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:53:58,832 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:53:58,840 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:53:58,840 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:58,840 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:53:58,840 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:53:58,840 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-27 03:53:58,841 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:53:58,841 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:58,841 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:53:58,841 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:58,841 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:53:58,841 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:58,873 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=423 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/3805a27beb40460c8f41a76d0384b78a 2023-04-27 03:53:59,306 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=423 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/c2e4ced10deb40c195c58e7d7ddbe05f 2023-04-27 03:53:59,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:59,662 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:53:59,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:59,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567699671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:59,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567699672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:59,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567699673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:59,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567699673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:59,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567699674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,725 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=423 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/130431b6f9124c44bdb0a0c69f830204 2023-04-27 03:53:59,732 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/3805a27beb40460c8f41a76d0384b78a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/3805a27beb40460c8f41a76d0384b78a 2023-04-27 03:53:59,737 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/3805a27beb40460c8f41a76d0384b78a, entries=150, sequenceid=423, filesize=11.9 K 2023-04-27 03:53:59,739 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/c2e4ced10deb40c195c58e7d7ddbe05f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/c2e4ced10deb40c195c58e7d7ddbe05f 2023-04-27 03:53:59,744 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/c2e4ced10deb40c195c58e7d7ddbe05f, entries=150, sequenceid=423, filesize=11.9 K 2023-04-27 03:53:59,745 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/130431b6f9124c44bdb0a0c69f830204 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/130431b6f9124c44bdb0a0c69f830204 2023-04-27 03:53:59,749 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/130431b6f9124c44bdb0a0c69f830204, entries=150, sequenceid=423, filesize=11.9 K 2023-04-27 03:53:59,750 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 910ms, sequenceid=423, compaction requested=true 2023-04-27 03:53:59,751 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:53:59,751 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:53:59,751 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:53:59,751 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:53:59,751 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,751 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:53:59,751 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:53:59,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:53:59,782 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-27 03:53:59,783 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:53:59,783 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:59,783 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:53:59,783 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:59,783 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:53:59,783 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:53:59,793 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,793 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:59,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:59,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:59,793 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:59,793 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:53:59,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:59,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:59,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:59,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:59,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:53:59,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18d0854d[Count = 0] remaining members to acquire global barrier 2023-04-27 03:53:59,797 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:53:59,797 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:59,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:59,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567699799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:59,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:59,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567699799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567699800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:59,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567699801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,823 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:59,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:59,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:59,824 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:53:59,824 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:53:59,824 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,824 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:53:59,824 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:53:59,854 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,854 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:53:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:59,854 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:59,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:53:59,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:59,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:53:59,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:53:59,857 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:53:59,857 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:53:59,857 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:53:59,857 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:53:59,857 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:53:59,877 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:59,877 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:59,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:59,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:53:59,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:53:59,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:59,877 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:59,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:59,878 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:53:59,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:59,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:59,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:59,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:53:59,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:53:59,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:59,879 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:53:59,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:53:59,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:59,909 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:53:59,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567699907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:53:59,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:59,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:59,909 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567699907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567699907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:53:59,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:53:59,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:53:59,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:53:59,909 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:53:59,910 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:53:59,910 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:53:59,909 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:53:59,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:53:59,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:53:59,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567699907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,910 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:59,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:53:59,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:53:59,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:53:59,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:59,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:53:59,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,010 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:00,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:00,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:00,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:00,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:00,014 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:00,014 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:00,014 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:00,014 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:00,015 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,015 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:00,039 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:00,039 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:00,039 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,039 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:00,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:00,040 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:00,040 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:00,040 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:00,040 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:00,040 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:00,041 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:00,041 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:00,041 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:00,041 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:54:00,041 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:54:00,041 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:00,041 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:00,041 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:00,041 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,041 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:00,041 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:00,085 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,085 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:00,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:00,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:00,085 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:00,086 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:00,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:00,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:00,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:00,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:00,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:00,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@605fade1[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:00,087 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:00,087 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:00,108 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:00,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:00,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:00,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:00,108 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,116 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:00,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:00,116 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:00,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:00,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567700111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:00,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567700111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:00,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567700112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:00,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567700112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,118 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,118 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:00,118 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:00,118 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:00,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:00,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:00,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:00,122 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:00,122 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:00,122 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:00,122 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:00,122 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:00,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:00,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:00,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:00,125 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:00,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:00,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,126 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:00,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:00,132 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:00,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:00,132 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:00,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:00,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:00,132 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:00,132 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:00,132 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:00,132 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:00,132 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:00,132 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,133 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:00,133 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:00,133 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,201 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/da32a867ba3c4ebf87b875e0fbc2d024 2023-04-27 03:54:00,233 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:00,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:00,235 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/9c248c7cd426479b914121e159b95380 2023-04-27 03:54:00,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:00,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:00,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:00,257 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:00,257 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:00,257 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:00,257 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:00,258 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,258 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:00,260 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:00,260 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:00,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:00,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:00,261 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,261 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:00,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:00,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:00,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:00,267 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:00,267 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:00,267 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:00,267 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:00,267 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:00,267 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:00,267 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:00,267 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:00,268 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:54:00,268 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:54:00,268 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:00,268 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:00,268 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:00,268 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,268 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:00,268 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:00,270 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,270 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:00,270 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:00,270 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:00,271 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/a4ee3583d0ba4a88b0b91e3d1e061238 2023-04-27 03:54:00,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:00,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:00,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:00,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75a3d264[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:00,273 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:00,273 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:00,274 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:00,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:00,275 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:00,275 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:00,275 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:00,276 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,276 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:00,278 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/da32a867ba3c4ebf87b875e0fbc2d024 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/da32a867ba3c4ebf87b875e0fbc2d024 2023-04-27 03:54:00,281 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,281 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:00,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:00,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:00,281 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:00,282 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:00,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:00,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:00,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:00,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:00,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:00,299 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/da32a867ba3c4ebf87b875e0fbc2d024, entries=200, sequenceid=438, filesize=14.3 K 2023-04-27 03:54:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:00,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:00,299 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:00,300 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:00,300 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:00,300 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:00,300 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:00,300 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/9c248c7cd426479b914121e159b95380 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/9c248c7cd426479b914121e159b95380 2023-04-27 03:54:00,301 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,301 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:00,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,302 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:00,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:00,303 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:00,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:00,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:00,304 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,306 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/9c248c7cd426479b914121e159b95380, entries=150, sequenceid=438, filesize=11.9 K 2023-04-27 03:54:00,307 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/a4ee3583d0ba4a88b0b91e3d1e061238 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/a4ee3583d0ba4a88b0b91e3d1e061238 2023-04-27 03:54:00,311 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,311 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:00,311 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:00,311 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,311 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:00,311 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:00,311 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:00,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:00,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:00,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:00,311 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,311 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:00,311 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:00,311 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:00,311 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:00,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:00,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:00,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:00,313 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/a4ee3583d0ba4a88b0b91e3d1e061238, entries=150, sequenceid=438, filesize=11.9 K 2023-04-27 03:54:00,313 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 531ms, sequenceid=438, compaction requested=true 2023-04-27 03:54:00,314 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:54:00,314 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:00,314 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-27 03:54:00,314 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-27 03:54:00,314 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:00,314 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:00,316 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 76272 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-27 03:54:00,316 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73832 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-27 03:54:00,316 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/A is initiating minor compaction (all files) 2023-04-27 03:54:00,316 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/B is initiating minor compaction (all files) 2023-04-27 03:54:00,316 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/A in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:00,316 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/B in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:00,316 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/b4312b8e1f904a1b9ff7cfe501e5d976, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/112c6f6dab5c438892f42025c696582e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/543aa6d428d043e1863d36d2e71cb9a1, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/1929a49af4d0450183b1a5907a45c23a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/3805a27beb40460c8f41a76d0384b78a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/da32a867ba3c4ebf87b875e0fbc2d024] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=74.5 K 2023-04-27 03:54:00,316 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/7f8bafed60a14718bd366d4b528b4894, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/06c006c3168d44b485137cf9e63579d8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/628e24b6f9e24bf8bd3d4cfaa28c4d8a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/f7296eef8d0c4e3095044897ad3b9baa, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/c2e4ced10deb40c195c58e7d7ddbe05f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/9c248c7cd426479b914121e159b95380] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=72.1 K 2023-04-27 03:54:00,319 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting b4312b8e1f904a1b9ff7cfe501e5d976, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1682567634598 2023-04-27 03:54:00,319 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 7f8bafed60a14718bd366d4b528b4894, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1682567634598 2023-04-27 03:54:00,319 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 06c006c3168d44b485137cf9e63579d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1682567634623 2023-04-27 03:54:00,319 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 112c6f6dab5c438892f42025c696582e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1682567634623 2023-04-27 03:54:00,320 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 628e24b6f9e24bf8bd3d4cfaa28c4d8a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1682567635771 2023-04-27 03:54:00,320 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 543aa6d428d043e1863d36d2e71cb9a1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1682567635771 2023-04-27 03:54:00,320 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting f7296eef8d0c4e3095044897ad3b9baa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1682567636903 2023-04-27 03:54:00,320 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 1929a49af4d0450183b1a5907a45c23a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1682567636903 2023-04-27 03:54:00,321 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting c2e4ced10deb40c195c58e7d7ddbe05f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=423, earliestPutTs=1682567637514 2023-04-27 03:54:00,321 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 3805a27beb40460c8f41a76d0384b78a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=423, earliestPutTs=1682567637514 2023-04-27 03:54:00,322 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 9c248c7cd426479b914121e159b95380, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682567639669 2023-04-27 03:54:00,322 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting da32a867ba3c4ebf87b875e0fbc2d024, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682567639669 2023-04-27 03:54:00,336 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#B#compaction#80 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:00,338 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#A#compaction#81 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:00,354 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/1ae831fd16c44b65bdac19667c5a28ab as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/1ae831fd16c44b65bdac19667c5a28ab 2023-04-27 03:54:00,360 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/A of 2b04f11bbacd80b8750d75eb2d0fbb35 into 1ae831fd16c44b65bdac19667c5a28ab(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:00,360 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:54:00,360 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/A, priority=10, startTime=1682567640314; duration=0sec 2023-04-27 03:54:00,360 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:00,360 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-27 03:54:00,362 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73798 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-27 03:54:00,362 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/C is initiating minor compaction (all files) 2023-04-27 03:54:00,362 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/C in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:00,362 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/81c2e0edd58b493cb31a31e653615a2f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/b2940039c8e645b2b3686905d2c7b596, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/6be0734d02c54fe4aa7a1ec89bed28de, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/79b4152c94d54c0eb69d65f6d01600f6, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/130431b6f9124c44bdb0a0c69f830204, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/a4ee3583d0ba4a88b0b91e3d1e061238] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=72.1 K 2023-04-27 03:54:00,363 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 81c2e0edd58b493cb31a31e653615a2f, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1682567634598 2023-04-27 03:54:00,363 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting b2940039c8e645b2b3686905d2c7b596, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1682567634623 2023-04-27 03:54:00,363 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 6be0734d02c54fe4aa7a1ec89bed28de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1682567635771 2023-04-27 03:54:00,364 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 79b4152c94d54c0eb69d65f6d01600f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1682567636903 2023-04-27 03:54:00,364 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 130431b6f9124c44bdb0a0c69f830204, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=423, earliestPutTs=1682567637514 2023-04-27 03:54:00,365 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting a4ee3583d0ba4a88b0b91e3d1e061238, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682567639669 2023-04-27 03:54:00,377 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#C#compaction#82 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:00,411 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:00,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:00,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:00,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:00,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:00,416 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:00,416 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:00,416 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:00,416 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:00,417 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,417 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:00,418 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,418 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:00,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:00,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:00,419 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,419 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:00,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:00,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:00,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:00,421 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:00,421 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:00,421 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:54:00,421 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-27 03:54:00,421 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:54:00,421 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:00,421 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:54:00,421 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:00,421 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:54:00,421 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:00,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:54:00,424 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:54:00,432 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/e0778c8917e343c9a51b06e8c12d26ef 2023-04-27 03:54:00,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:00,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567700439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:00,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567700440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:00,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567700442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:00,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567700442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,458 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/b861edb5cd7545f0bfb4e2bf7eb9114a 2023-04-27 03:54:00,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:00,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567700543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:00,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567700543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,552 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:00,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567700551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,552 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:00,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567700551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,750 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/cf3a2c1184e84e33b572c2d139906f25 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/cf3a2c1184e84e33b572c2d139906f25 2023-04-27 03:54:00,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:00,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567700748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:00,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567700748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:00,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567700753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,759 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:00,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567700758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,770 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/B of 2b04f11bbacd80b8750d75eb2d0fbb35 into cf3a2c1184e84e33b572c2d139906f25(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:00,770 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:54:00,770 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/B, priority=10, startTime=1682567640314; duration=0sec 2023-04-27 03:54:00,770 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:00,790 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/603695e6d54444c09a4c06e1b2d4b63c as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/603695e6d54444c09a4c06e1b2d4b63c 2023-04-27 03:54:00,796 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/C of 2b04f11bbacd80b8750d75eb2d0fbb35 into 603695e6d54444c09a4c06e1b2d4b63c(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:00,796 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:54:00,796 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/C, priority=10, startTime=1682567640314; duration=0sec 2023-04-27 03:54:00,796 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:00,878 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/5882ea23cef7451f9ce867e233578fdd 2023-04-27 03:54:00,885 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/e0778c8917e343c9a51b06e8c12d26ef as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/e0778c8917e343c9a51b06e8c12d26ef 2023-04-27 03:54:00,894 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/e0778c8917e343c9a51b06e8c12d26ef, entries=150, sequenceid=456, filesize=11.9 K 2023-04-27 03:54:00,896 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/b861edb5cd7545f0bfb4e2bf7eb9114a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/b861edb5cd7545f0bfb4e2bf7eb9114a 2023-04-27 03:54:00,903 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/b861edb5cd7545f0bfb4e2bf7eb9114a, entries=150, sequenceid=456, filesize=11.9 K 2023-04-27 03:54:00,904 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/5882ea23cef7451f9ce867e233578fdd as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/5882ea23cef7451f9ce867e233578fdd 2023-04-27 03:54:00,913 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/5882ea23cef7451f9ce867e233578fdd, entries=150, sequenceid=456, filesize=11.9 K 2023-04-27 03:54:00,914 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 493ms, sequenceid=456, compaction requested=false 2023-04-27 03:54:00,914 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:54:00,914 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:00,914 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:00,914 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:00,914 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,916 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:00,916 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:00,917 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,917 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:00,918 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:00,918 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:00,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:00,919 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:00,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37f08b6b[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:00,919 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:00,921 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:00,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:00,921 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:00,921 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:00,921 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:00,922 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,922 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:00,923 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,923 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:00,923 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:00,923 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:00,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:00,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:00,926 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:00,926 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:00,926 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:00,927 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:00,927 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:00,928 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,928 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:00,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,929 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:00,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:00,930 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,931 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:00,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,934 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,934 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:00,934 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:00,934 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:00,934 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:00,935 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:00,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:00,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:00,934 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:00,935 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:00,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:00,935 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:00,935 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:00,935 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:00,935 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:00,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:01,035 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:01,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:01,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:01,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:01,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:01,053 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:01,053 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:01,053 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:01,053 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:01,053 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,053 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:01,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:54:01,055 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-27 03:54:01,055 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:54:01,055 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:01,055 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:54:01,055 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:01,055 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:54:01,055 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:01,056 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,056 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:01,056 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,056 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:01,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:01,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:01,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:01,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:01,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:01,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:01,082 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:01,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:01,083 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:01,083 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:01,083 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:01,092 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:01,092 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:01,092 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:01,092 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:54:01,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:01,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567701083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,092 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:54:01,093 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:01,093 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:01,093 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:01,093 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:01,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567701089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:01,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567701092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:01,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567701093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,104 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:01,104 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:01,107 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:01,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:01,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:01,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:01,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:01,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:01,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b169a35[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:01,109 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:01,109 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,111 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,111 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:01,111 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:01,111 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:01,112 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,112 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:01,113 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:01,113 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:01,113 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,113 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:01,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:01,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:01,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:01,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:01,117 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:01,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:01,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:01,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:01,117 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:01,119 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,120 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:01,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,120 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:01,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,121 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:01,123 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:01,126 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:01,126 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:01,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:01,126 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:01,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:01,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:01,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:01,127 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:01,127 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:01,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:01,128 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,128 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:01,128 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:01,128 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,128 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,128 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,128 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:01,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567701194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:01,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567701194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:01,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567701197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:01,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567701197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,227 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:01,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:01,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:01,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:01,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:01,231 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:01,231 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:01,231 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:01,231 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:01,231 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,231 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:01,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:01,233 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:01,233 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,233 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:01,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:01,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:01,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:01,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:01,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:01,234 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:01,234 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:01,234 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:01,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:01,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:01,236 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:01,236 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:54:01,236 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:54:01,236 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:01,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:01,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:01,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:01,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:01,237 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,237 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:01,238 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,238 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:01,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:01,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36bb0a13[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:01,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:01,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,242 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,242 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:01,242 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:01,242 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:01,242 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,242 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:01,243 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:01,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:01,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:01,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:01,246 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:01,246 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:01,246 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:01,246 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:01,246 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:01,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:01,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:01,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:01,249 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,250 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,255 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:01,255 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:01,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:01,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:01,255 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,255 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:01,255 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:01,255 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,255 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,255 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,256 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,256 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:01,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:01,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:01,256 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:01,256 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:01,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:01,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:01,356 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:01,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:01,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:01,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:01,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:01,376 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:01,376 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:01,376 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:01,376 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:01,377 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,377 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:01,379 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,379 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:01,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:01,380 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,380 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:01,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:01,382 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:01,382 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:01,382 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:01,382 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:01,382 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:01,382 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:01,384 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:01,384 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:01,384 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:01,384 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:54:01,384 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:54:01,384 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:01,384 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:01,384 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:01,385 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,385 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:01,385 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:01,386 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,386 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:01,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:01,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:01,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d60de52[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:01,390 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:01,390 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,393 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,393 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:01,393 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:01,393 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:01,395 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,395 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:01,395 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:01,395 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:01,396 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:01,396 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:01,396 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:01,396 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:01,396 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:01,396 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:01,396 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:01,397 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:01,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:01,398 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,398 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:01,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:01,398 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:01,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:01,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:01,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:01,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567701397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:01,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567701397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:01,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567701400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,401 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:01,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:01,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:01,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567701402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,404 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,404 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:01,404 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:01,404 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:01,404 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:01,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:01,404 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:01,404 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:01,404 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:01,404 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:01,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:01,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:01,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:01,404 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:01,405 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:01,404 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:01,505 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:01,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:01,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:01,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:01,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:01,510 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:01,510 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:01,510 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:01,510 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:01,510 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,510 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:01,512 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:01,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:01,512 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:01,513 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,513 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:01,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:01,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:01,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:01,514 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:01,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:01,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:01,516 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:01,516 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:01,516 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:01,524 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:01,524 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:01,524 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:01,524 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:54:01,524 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:54:01,524 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:01,524 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:01,525 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:01,525 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,525 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:01,525 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:01,527 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,527 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:01,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:01,528 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,528 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:01,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:01,529 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/fabcd20dbdac4ebc88ad24b9aa03f1f4 2023-04-27 03:54:01,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:01,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:01,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:01,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:01,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e78ad46[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:01,530 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:01,531 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,532 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,532 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:01,532 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:01,532 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:01,532 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,532 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:01,533 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:01,534 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:01,534 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:01,534 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:01,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:01,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:01,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:01,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:01,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:01,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:01,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:01,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:01,538 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:01,538 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:01,538 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:01,538 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:01,538 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:01,541 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:01,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:01,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:01,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:01,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:01,542 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,542 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:01,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:01,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:01,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:01,543 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:01,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,545 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,547 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,547 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:01,547 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:01,547 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:01,547 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:01,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:01,547 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:01,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:01,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:01,547 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,548 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,548 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:01,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:01,548 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:01,548 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:01,548 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,555 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/4c2319f7090043c9870943ed95593ab3 2023-04-27 03:54:01,590 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/213b218d714d45c2b71e8e577631be7c 2023-04-27 03:54:01,595 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/fabcd20dbdac4ebc88ad24b9aa03f1f4 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/fabcd20dbdac4ebc88ad24b9aa03f1f4 2023-04-27 03:54:01,603 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/fabcd20dbdac4ebc88ad24b9aa03f1f4, entries=150, sequenceid=475, filesize=11.9 K 2023-04-27 03:54:01,606 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/4c2319f7090043c9870943ed95593ab3 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/4c2319f7090043c9870943ed95593ab3 2023-04-27 03:54:01,615 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/4c2319f7090043c9870943ed95593ab3, entries=150, sequenceid=475, filesize=11.9 K 2023-04-27 03:54:01,616 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/213b218d714d45c2b71e8e577631be7c as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/213b218d714d45c2b71e8e577631be7c 2023-04-27 03:54:01,622 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/213b218d714d45c2b71e8e577631be7c, entries=150, sequenceid=475, filesize=11.9 K 2023-04-27 03:54:01,623 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 568ms, sequenceid=475, compaction requested=true 2023-04-27 03:54:01,623 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:54:01,624 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:01,624 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:01,624 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:01,624 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:01,624 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:01,630 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:01,630 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/B is initiating minor compaction (all files) 2023-04-27 03:54:01,631 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/B in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:01,631 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/cf3a2c1184e84e33b572c2d139906f25, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/b861edb5cd7545f0bfb4e2bf7eb9114a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/4c2319f7090043c9870943ed95593ab3] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=36.6 K 2023-04-27 03:54:01,631 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting cf3a2c1184e84e33b572c2d139906f25, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682567639669 2023-04-27 03:54:01,632 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting b861edb5cd7545f0bfb4e2bf7eb9114a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1682567639798 2023-04-27 03:54:01,632 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 4c2319f7090043c9870943ed95593ab3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1682567640424 2023-04-27 03:54:01,636 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:01,636 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/A is initiating minor compaction (all files) 2023-04-27 03:54:01,636 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/A in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:01,636 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/1ae831fd16c44b65bdac19667c5a28ab, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/e0778c8917e343c9a51b06e8c12d26ef, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/fabcd20dbdac4ebc88ad24b9aa03f1f4] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=36.6 K 2023-04-27 03:54:01,637 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 1ae831fd16c44b65bdac19667c5a28ab, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682567639669 2023-04-27 03:54:01,638 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting e0778c8917e343c9a51b06e8c12d26ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1682567639798 2023-04-27 03:54:01,639 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting fabcd20dbdac4ebc88ad24b9aa03f1f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1682567640424 2023-04-27 03:54:01,645 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#B#compaction#89 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:01,648 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#A#compaction#90 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:01,648 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:01,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:01,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:01,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:01,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:01,664 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:01,664 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:01,664 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:01,664 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:01,664 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,664 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:01,666 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:01,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:01,666 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:01,667 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,667 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:01,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:01,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:01,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:01,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:01,676 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:01,676 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:01,676 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:01,687 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/aea24cb3ae7540fd9a2f74ce8d2ade10 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/aea24cb3ae7540fd9a2f74ce8d2ade10 2023-04-27 03:54:01,687 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:01,687 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:01,688 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:01,688 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:54:01,688 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-27 03:54:01,688 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:54:01,688 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:01,688 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:54:01,688 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:01,688 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:54:01,689 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:01,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:54:01,690 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:54:01,708 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/283c1f28e6104b29a56d916a56a725b3 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/283c1f28e6104b29a56d916a56a725b3 2023-04-27 03:54:01,708 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/B of 2b04f11bbacd80b8750d75eb2d0fbb35 into aea24cb3ae7540fd9a2f74ce8d2ade10(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:01,708 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:54:01,708 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/B, priority=13, startTime=1682567641624; duration=0sec 2023-04-27 03:54:01,709 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:01,709 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:01,711 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:01,711 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/C is initiating minor compaction (all files) 2023-04-27 03:54:01,711 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/C in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:01,711 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/603695e6d54444c09a4c06e1b2d4b63c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/5882ea23cef7451f9ce867e233578fdd, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/213b218d714d45c2b71e8e577631be7c] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=36.6 K 2023-04-27 03:54:01,716 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 603695e6d54444c09a4c06e1b2d4b63c, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682567639669 2023-04-27 03:54:01,717 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 5882ea23cef7451f9ce867e233578fdd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1682567639798 2023-04-27 03:54:01,717 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 213b218d714d45c2b71e8e577631be7c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1682567640424 2023-04-27 03:54:01,720 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/A of 2b04f11bbacd80b8750d75eb2d0fbb35 into 283c1f28e6104b29a56d916a56a725b3(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:01,720 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:54:01,720 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/A, priority=13, startTime=1682567641623; duration=0sec 2023-04-27 03:54:01,721 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:01,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:01,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567701716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:01,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567701716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:01,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567701718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:01,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567701719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:01,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567701719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,728 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#C#compaction#92 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:01,730 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/88c7df9e43d14eae97f20c13cc23b791 2023-04-27 03:54:01,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:01,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:01,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567701824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567701824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:01,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567701828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,831 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:01,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567701828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:01,831 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:01,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567701828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567702027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567702031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567702031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567702033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567702033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,163 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/8cf19a383ff1439490d986086ab3b6c5 2023-04-27 03:54:02,163 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/9a3f126351654b4a94f362645a0b311d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/9a3f126351654b4a94f362645a0b311d 2023-04-27 03:54:02,174 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/C of 2b04f11bbacd80b8750d75eb2d0fbb35 into 9a3f126351654b4a94f362645a0b311d(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:02,174 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:54:02,174 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/C, priority=13, startTime=1682567641624; duration=0sec 2023-04-27 03:54:02,174 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:02,183 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/dad49b6745ab416b8a05e7d2ae5f7871 2023-04-27 03:54:02,193 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/88c7df9e43d14eae97f20c13cc23b791 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/88c7df9e43d14eae97f20c13cc23b791 2023-04-27 03:54:02,198 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/88c7df9e43d14eae97f20c13cc23b791, entries=150, sequenceid=490, filesize=11.9 K 2023-04-27 03:54:02,199 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/8cf19a383ff1439490d986086ab3b6c5 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/8cf19a383ff1439490d986086ab3b6c5 2023-04-27 03:54:02,204 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/8cf19a383ff1439490d986086ab3b6c5, entries=150, sequenceid=490, filesize=11.9 K 2023-04-27 03:54:02,205 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/dad49b6745ab416b8a05e7d2ae5f7871 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/dad49b6745ab416b8a05e7d2ae5f7871 2023-04-27 03:54:02,209 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/dad49b6745ab416b8a05e7d2ae5f7871, entries=150, sequenceid=490, filesize=11.9 K 2023-04-27 03:54:02,211 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 523ms, sequenceid=490, compaction requested=false 2023-04-27 03:54:02,211 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:54:02,211 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:02,211 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:02,211 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:02,211 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,211 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:02,211 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:02,216 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:02,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:02,216 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,217 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,217 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:02,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:02,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7120458e[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:02,219 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:02,219 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,222 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,222 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:02,228 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,228 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:02,228 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:02,228 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:02,230 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,230 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:02,230 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:02,230 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:02,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:02,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:02,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:02,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:02,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:02,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:02,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:02,233 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:02,233 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:02,233 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:02,233 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:02,233 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:02,235 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,235 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:02,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,235 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:02,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:02,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:02,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,236 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:02,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:02,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,242 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,242 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,242 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,242 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,242 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,242 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,242 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,242 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:02,242 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:02,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:02,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:02,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:02,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:02,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:02,242 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:02,243 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:02,243 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:02,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:02,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:02,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:54:02,336 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-27 03:54:02,336 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:54:02,336 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:02,336 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:54:02,336 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:02,336 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:54:02,336 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:02,343 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:02,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:02,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:02,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:02,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:02,352 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:02,352 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:02,352 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:02,352 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:02,352 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,353 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567702347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567702351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567702352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567702353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567702354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,357 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:02,357 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:02,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:02,358 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,358 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:02,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:02,364 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:02,364 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:02,364 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:02,364 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:02,364 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:02,368 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:02,368 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:02,368 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:02,368 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:54:02,368 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:54:02,368 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:02,368 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:02,368 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:02,368 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,368 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:02,368 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:02,370 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,370 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:02,371 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,371 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:02,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:02,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:02,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:02,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:02,372 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:02,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fa5322b[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:02,372 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,373 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,374 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:02,374 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:02,374 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:02,374 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,374 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:02,375 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,375 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:02,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:02,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:02,375 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:02,375 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:02,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:02,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:02,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:02,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:02,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:02,378 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:02,378 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:02,378 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:02,378 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:02,378 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:02,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:02,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:02,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:02,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:02,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:02,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:02,390 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:02,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,391 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:02,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,394 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,394 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,394 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,394 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,394 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,394 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,394 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:02,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:02,394 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:02,394 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:02,394 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:02,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:02,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:02,395 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:02,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:02,394 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:02,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:02,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:02,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567702455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567702456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567702456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567702457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567702457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,495 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:02,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:02,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:02,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:02,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:02,499 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:02,499 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:02,499 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:02,499 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:02,500 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,500 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:02,501 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:02,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:02,502 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,502 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:02,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:02,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:02,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:02,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:02,503 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:02,503 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:02,503 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:02,503 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:02,503 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:02,504 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:02,504 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:54:02,504 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:54:02,504 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:02,504 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:02,504 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:02,504 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,504 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:02,504 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:02,514 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:02,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:02,515 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:02,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:02,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:02,520 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,520 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:02,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:02,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@446cedc3[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:02,520 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:02,520 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,523 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,523 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:02,523 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:02,523 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:02,523 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,523 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:02,525 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:02,525 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,525 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:02,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:02,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:02,525 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:02,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:02,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:02,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:02,529 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:02,530 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:02,530 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:02,530 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:02,530 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:02,532 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:02,532 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,532 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,532 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:02,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:02,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:02,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,534 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:02,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:02,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:02,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:02,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:02,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:02,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:02,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:02,537 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:02,537 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:02,536 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:02,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:02,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:02,637 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:02,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:02,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:02,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:02,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:02,644 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:02,644 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:02,644 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:02,644 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:02,644 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,645 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,647 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,647 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:02,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:02,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:02,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,648 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,648 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:02,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:02,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:02,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:02,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:02,649 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:02,649 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:02,649 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:02,649 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:02,649 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:02,650 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:02,650 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:54:02,650 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:54:02,650 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:02,650 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:02,650 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:02,650 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,650 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:02,650 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:02,651 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:02,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:02,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:02,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:02,653 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,654 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,654 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:02,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:02,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:02,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f556a95[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:02,654 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:02,655 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,656 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,656 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:02,656 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:02,656 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:02,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567702658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,660 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,660 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:02,661 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,661 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:02,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,661 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:02,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:02,661 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:02,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:02,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:02,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567702660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567702660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567702660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567702660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:02,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:02,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:02,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:02,664 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:02,664 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:02,664 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:02,665 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:02,665 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:02,670 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,670 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:02,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:02,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,670 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:02,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:02,671 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:02,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:02,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:02,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,672 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:02,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,675 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,675 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,675 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:02,675 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:02,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:02,675 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:02,675 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:02,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:02,675 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:02,676 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:02,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:02,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:02,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:02,675 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,676 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,676 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,676 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:02,750 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/f457f9983753480b89a8c64fae5c10d5 2023-04-27 03:54:02,776 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:02,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:02,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:02,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:02,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:02,786 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:02,786 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:02,786 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:02,786 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:02,788 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/b2505b26be454ce0ab2f17374a520126 2023-04-27 03:54:02,788 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,788 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,789 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:02,789 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:02,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:02,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,790 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,791 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:02,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:02,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,791 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:02,792 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:02,792 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:02,792 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:02,792 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:54:02,792 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:54:02,792 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,793 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:02,793 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:02,794 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,794 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:02,795 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,795 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:02,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:02,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:02,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c2a5ec4[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:02,800 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:02,800 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,805 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,805 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:02,805 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:02,805 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:02,805 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,805 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:02,813 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,813 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:02,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:02,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:02,813 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:02,813 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:02,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:02,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:02,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:02,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:02,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:02,816 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:02,816 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:02,816 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:02,816 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:02,816 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:02,818 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,818 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:02,818 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,819 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:02,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:02,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,825 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:02,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,827 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:02,827 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:02,827 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:02,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:02,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:02,827 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:02,827 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:02,827 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,827 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,827 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,827 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:02,827 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:02,827 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,828 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:02,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:02,928 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:02,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:02,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:02,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:02,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:02,931 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:02,931 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:02,931 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:02,931 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:02,932 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,932 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,934 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:02,934 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:02,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:02,934 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,935 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:02,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:02,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,935 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:02,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:02,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:02,936 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:02,936 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:02,936 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:02,936 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:02,936 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:02,936 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:02,936 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:54:02,936 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:54:02,936 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:02,936 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:02,936 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:02,936 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,936 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:02,937 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:02,938 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,938 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:02,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:02,939 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,939 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:02,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:02,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:02,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:02,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:02,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a280300[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:02,940 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:02,940 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,942 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,942 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:02,942 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:02,942 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:02,942 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,942 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:02,943 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,943 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:02,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:02,943 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:02,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:02,943 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:02,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:02,946 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:02,946 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:02,946 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:02,946 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:02,946 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:02,950 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,950 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:02,951 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:02,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:02,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:02,951 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,953 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:02,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,956 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,956 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,956 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:02,956 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,956 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,956 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:02,956 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:02,956 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:02,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:02,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:02,957 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:02,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:02,957 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:02,957 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:02,957 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:02,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:02,960 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567702960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567702963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567702963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:02,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567702963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:02,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567702964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,057 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:03,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:03,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:03,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:03,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:03,060 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:03,060 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:03,060 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:03,060 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:03,061 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:03,061 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:03,062 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:03,062 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:03,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:03,062 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,062 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:03,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:03,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:03,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:03,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:03,063 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:03,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:03,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:03,064 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:03,064 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:03,064 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:03,064 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:03,064 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:03,064 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:03,064 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:54:03,064 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:54:03,064 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:03,064 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:03,064 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:03,064 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,064 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:03,064 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:03,065 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,065 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:03,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:03,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:03,066 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:03,066 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:03,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:03,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:03,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:03,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:03,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:03,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@909f173[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:03,067 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:03,067 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:03,069 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:03,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:03,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:03,069 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:03,070 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:03,070 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:03,070 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,070 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:03,071 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,071 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:03,071 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:03,071 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:03,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:03,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:03,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:03,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:03,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:03,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:03,074 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:03,075 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:03,075 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:03,075 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:03,075 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:03,076 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:03,077 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:03,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:03,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:03,078 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:03,078 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:03,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:03,079 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:03,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:03,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:03,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:03,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:03,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:03,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:03,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,082 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,082 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:03,082 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:03,082 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,082 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:03,082 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:03,082 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:03,082 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:03,082 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:03,082 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:03,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:03,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:03,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:03,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:03,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:03,083 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:03,083 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:03,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:03,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:03,183 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:03,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:03,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:03,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:03,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:03,187 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:03,187 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:03,187 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:03,187 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:03,187 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:03,187 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:03,189 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,189 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:03,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:03,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:03,190 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,190 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:03,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:03,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:03,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:03,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:03,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:03,191 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:03,191 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:03,191 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:03,191 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:03,191 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:03,191 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:03,191 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:03,191 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:03,191 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:54:03,191 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:54:03,192 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:03,192 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:03,192 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:03,192 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,192 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:03,192 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:03,193 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,193 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:03,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:03,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:03,193 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:03,193 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:03,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:03,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:03,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:03,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:03,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:03,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dcc3305[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:03,195 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:03,195 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:03,196 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:03,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:03,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:03,196 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:03,196 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:03,196 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,196 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:03,196 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:03,201 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:03,201 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:03,202 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:03,202 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:03,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:03,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:03,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:03,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:03,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:03,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:03,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:03,205 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:03,206 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:03,206 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:03,206 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:03,206 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:03,207 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:03,207 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:03,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:03,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:03,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:03,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:03,207 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:03,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:03,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:03,208 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:03,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:03,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:03,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:03,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:03,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:03,210 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/19c53a06a75a43fe8fe580b69d2686c2 2023-04-27 03:54:03,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,210 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:03,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,215 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:03,215 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:03,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:03,215 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:03,215 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:03,215 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:03,215 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:03,215 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,215 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:03,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:03,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:03,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:03,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:03,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:03,215 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:03,215 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:03,215 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:03,216 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:03,216 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/f457f9983753480b89a8c64fae5c10d5 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/f457f9983753480b89a8c64fae5c10d5 2023-04-27 03:54:03,221 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/f457f9983753480b89a8c64fae5c10d5, entries=250, sequenceid=512, filesize=16.7 K 2023-04-27 03:54:03,222 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/b2505b26be454ce0ab2f17374a520126 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/b2505b26be454ce0ab2f17374a520126 2023-04-27 03:54:03,228 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/b2505b26be454ce0ab2f17374a520126, entries=150, sequenceid=512, filesize=11.9 K 2023-04-27 03:54:03,229 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/19c53a06a75a43fe8fe580b69d2686c2 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/19c53a06a75a43fe8fe580b69d2686c2 2023-04-27 03:54:03,233 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/19c53a06a75a43fe8fe580b69d2686c2, entries=150, sequenceid=512, filesize=11.9 K 2023-04-27 03:54:03,234 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 898ms, sequenceid=512, compaction requested=true 2023-04-27 03:54:03,235 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:54:03,235 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:03,235 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:03,235 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:03,235 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:03,235 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:03,236 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42460 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:03,236 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/A is initiating minor compaction (all files) 2023-04-27 03:54:03,237 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/A in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:03,237 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:03,237 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/283c1f28e6104b29a56d916a56a725b3, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/88c7df9e43d14eae97f20c13cc23b791, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/f457f9983753480b89a8c64fae5c10d5] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=41.5 K 2023-04-27 03:54:03,237 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/B is initiating minor compaction (all files) 2023-04-27 03:54:03,237 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/B in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:03,237 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/aea24cb3ae7540fd9a2f74ce8d2ade10, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/8cf19a383ff1439490d986086ab3b6c5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/b2505b26be454ce0ab2f17374a520126] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=36.7 K 2023-04-27 03:54:03,237 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 283c1f28e6104b29a56d916a56a725b3, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1682567640424 2023-04-27 03:54:03,237 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting aea24cb3ae7540fd9a2f74ce8d2ade10, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1682567640424 2023-04-27 03:54:03,238 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 88c7df9e43d14eae97f20c13cc23b791, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1682567641069 2023-04-27 03:54:03,238 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 8cf19a383ff1439490d986086ab3b6c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1682567641069 2023-04-27 03:54:03,238 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting f457f9983753480b89a8c64fae5c10d5, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1682567641705 2023-04-27 03:54:03,238 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting b2505b26be454ce0ab2f17374a520126, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1682567641716 2023-04-27 03:54:03,245 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#B#compaction#98 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:03,247 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#A#compaction#99 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:03,259 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/2b40773950314450b492f792e17700dc as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/2b40773950314450b492f792e17700dc 2023-04-27 03:54:03,266 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/B of 2b04f11bbacd80b8750d75eb2d0fbb35 into 2b40773950314450b492f792e17700dc(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:03,266 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/feccbc6f3c134a6587a74e53afff3d32 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/feccbc6f3c134a6587a74e53afff3d32 2023-04-27 03:54:03,266 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:54:03,266 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/B, priority=13, startTime=1682567643235; duration=0sec 2023-04-27 03:54:03,267 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:03,267 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:03,268 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:03,268 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 2b04f11bbacd80b8750d75eb2d0fbb35/C is initiating minor compaction (all files) 2023-04-27 03:54:03,268 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2b04f11bbacd80b8750d75eb2d0fbb35/C in TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:03,268 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/9a3f126351654b4a94f362645a0b311d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/dad49b6745ab416b8a05e7d2ae5f7871, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/19c53a06a75a43fe8fe580b69d2686c2] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp, totalSize=36.7 K 2023-04-27 03:54:03,269 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 9a3f126351654b4a94f362645a0b311d, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1682567640424 2023-04-27 03:54:03,269 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting dad49b6745ab416b8a05e7d2ae5f7871, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1682567641069 2023-04-27 03:54:03,270 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 19c53a06a75a43fe8fe580b69d2686c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1682567641716 2023-04-27 03:54:03,290 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/A of 2b04f11bbacd80b8750d75eb2d0fbb35 into feccbc6f3c134a6587a74e53afff3d32(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:03,291 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:54:03,291 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/A, priority=13, startTime=1682567643235; duration=0sec 2023-04-27 03:54:03,291 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:03,295 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 2b04f11bbacd80b8750d75eb2d0fbb35#C#compaction#100 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:03,316 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:03,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:03,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:03,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:03,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:03,319 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:03,319 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:03,319 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:03,319 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:03,320 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:03,320 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:03,321 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:03,321 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:03,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:03,321 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,321 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:03,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:03,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:03,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:03,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:03,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:03,322 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:03,322 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:03,322 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:03,322 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:03,322 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:03,322 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:03,322 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:03,323 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:54:03,323 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-27 03:54:03,323 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:54:03,323 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:03,323 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:54:03,323 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:03,323 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:54:03,323 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:03,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:54:03,467 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:54:03,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:03,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567703479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:03,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567703480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:03,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567703482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:03,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567703483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:03,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567703483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:03,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567703583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:03,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567703584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:03,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567703584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:03,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567703586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:03,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567703586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,707 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/5a68a0a2239b400db5258f7dc9bbbeda as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/5a68a0a2239b400db5258f7dc9bbbeda 2023-04-27 03:54:03,713 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2b04f11bbacd80b8750d75eb2d0fbb35/C of 2b04f11bbacd80b8750d75eb2d0fbb35 into 5a68a0a2239b400db5258f7dc9bbbeda(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:03,713 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:54:03,713 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35., storeName=2b04f11bbacd80b8750d75eb2d0fbb35/C, priority=13, startTime=1682567643235; duration=0sec 2023-04-27 03:54:03,714 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:03,740 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/6515501824c643a2b3f45f34908a895c 2023-04-27 03:54:03,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:03,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567703787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:03,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567703787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:03,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567703787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:03,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567703787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:03,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:03,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567703788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:04,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567704089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:04,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567704089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:04,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567704089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:04,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567704091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,092 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:04,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567704091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,153 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/4762cebad68b48d7bf262bd61b7e7ab5 2023-04-27 03:54:04,173 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/148c63bb1cfa4a7cb11e73e8213dc394 2023-04-27 03:54:04,179 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/6515501824c643a2b3f45f34908a895c as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/6515501824c643a2b3f45f34908a895c 2023-04-27 03:54:04,185 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/6515501824c643a2b3f45f34908a895c, entries=150, sequenceid=527, filesize=11.9 K 2023-04-27 03:54:04,186 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/4762cebad68b48d7bf262bd61b7e7ab5 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/4762cebad68b48d7bf262bd61b7e7ab5 2023-04-27 03:54:04,192 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/4762cebad68b48d7bf262bd61b7e7ab5, entries=150, sequenceid=527, filesize=11.9 K 2023-04-27 03:54:04,194 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/148c63bb1cfa4a7cb11e73e8213dc394 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/148c63bb1cfa4a7cb11e73e8213dc394 2023-04-27 03:54:04,199 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/148c63bb1cfa4a7cb11e73e8213dc394, entries=150, sequenceid=527, filesize=11.9 K 2023-04-27 03:54:04,200 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 877ms, sequenceid=527, compaction requested=false 2023-04-27 03:54:04,200 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:54:04,200 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:04,200 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:04,200 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:04,200 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,200 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:04,200 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:04,203 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,203 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:04,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:04,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:04,203 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:04,203 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:04,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:04,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:04,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:04,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:04,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:04,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35e8806[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:04,205 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:04,205 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:04,206 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:04,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:04,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:04,206 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:04,206 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:04,206 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:04,206 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,206 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:04,207 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:04,207 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:04,207 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,207 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:04,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:04,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:04,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:04,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:04,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:04,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:04,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:04,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:04,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:04,210 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:04,210 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:04,210 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:04,210 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:04,210 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:04,212 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:04,212 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:04,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:04,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:04,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:04,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:04,212 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:04,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:04,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:04,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:04,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:04,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:04,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:04,213 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:04,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:04,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,214 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:04,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:04,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,218 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:04,218 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:04,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:04,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:04,218 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,218 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:04,219 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:04,219 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,219 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:04,219 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:04,219 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:04,219 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:04,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:04,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:04,219 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:04,219 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:04,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:04,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:04,319 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:04,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:04,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:04,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:04,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:04,323 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:04,323 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:04,323 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:04,323 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:04,323 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:04,323 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:04,324 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:04,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:04,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:04,325 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:04,325 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,325 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:04,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:04,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:04,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:04,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:04,326 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:04,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:04,326 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:04,326 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:04,326 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:04,326 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:04,326 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:04,326 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:04,326 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. started... 2023-04-27 03:54:04,327 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-27 03:54:04,327 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:54:04,327 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:04,327 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:54:04,327 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:04,327 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:54:04,327 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:04,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:54:04,594 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. as already flushing 2023-04-27 03:54:04,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:04,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567704597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:04,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567704600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:04,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567704601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:04,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567704601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:04,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567704602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,621 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6aad2e18 to 127.0.0.1:55020 2023-04-27 03:54:04,621 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:04,621 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6fef7a21 to 127.0.0.1:55020 2023-04-27 03:54:04,622 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:04,623 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49a45b6d to 127.0.0.1:55020 2023-04-27 03:54:04,623 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:04,626 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cd3abbd to 127.0.0.1:55020 2023-04-27 03:54:04,626 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:04,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:04,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567704703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:04,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567704704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:04,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567704705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:04,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567704705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:04,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567704706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,736 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/a83217b03c34442c922c2994d71440ac 2023-04-27 03:54:04,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:04,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:04,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567704905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567704905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:04,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567704906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:04,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567704908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:04,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:04,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567704908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:05,147 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/7ec0137e100c47f9a01988d00d7a3959 2023-04-27 03:54:05,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:05,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:05,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50850 deadline: 1682567705207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:05,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50862 deadline: 1682567705207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:05,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:05,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50874 deadline: 1682567705208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:05,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:05,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50852 deadline: 1682567705210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:05,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:05,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50928 deadline: 1682567705210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:05,558 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/fe04b330d3e245dda8889fb4db548538 2023-04-27 03:54:05,562 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/a83217b03c34442c922c2994d71440ac as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/a83217b03c34442c922c2994d71440ac 2023-04-27 03:54:05,566 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/a83217b03c34442c922c2994d71440ac, entries=150, sequenceid=547, filesize=11.9 K 2023-04-27 03:54:05,567 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/7ec0137e100c47f9a01988d00d7a3959 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/7ec0137e100c47f9a01988d00d7a3959 2023-04-27 03:54:05,571 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/7ec0137e100c47f9a01988d00d7a3959, entries=150, sequenceid=547, filesize=11.9 K 2023-04-27 03:54:05,571 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/fe04b330d3e245dda8889fb4db548538 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/fe04b330d3e245dda8889fb4db548538 2023-04-27 03:54:05,575 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/fe04b330d3e245dda8889fb4db548538, entries=150, sequenceid=547, filesize=11.9 K 2023-04-27 03:54:05,576 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 1250ms, sequenceid=547, compaction requested=true 2023-04-27 03:54:05,576 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:54:05,576 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:05,576 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:05,576 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:05,576 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:05,576 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:05,577 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:05,578 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:05,578 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:05,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:05,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:05,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:05,578 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:05,579 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:05,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:05,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:05,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:05,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:05,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:05,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:05,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6904ab6e[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:05,580 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:05,580 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:05,581 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:05,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:05,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:05,581 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:05,581 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:05,581 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:05,581 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:05,581 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:05,582 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:05,582 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:05,582 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:05,582 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:05,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:05,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:05,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:05,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:05,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:05,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:05,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:05,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:05,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:05,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:05,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:05,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:05,585 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:05,585 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:05,585 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:05,585 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:05,585 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:05,587 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:05,587 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:05,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:05,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:05,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:05,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:05,587 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:05,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:05,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:05,587 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:05,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:05,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:05,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:05,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:05,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:05,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:05,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:05,588 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:05,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:05,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:05,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:05,590 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:05,590 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:05,590 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:05,590 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:05,590 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:05,590 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:05,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:05,590 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:05,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:05,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:05,590 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:05,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:05,591 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:05,591 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:05,591 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:05,591 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:05,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:05,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:05,691 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:05,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:05,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:54:05,710 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-27 03:54:05,710 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fea27db to 127.0.0.1:55020 2023-04-27 03:54:05,711 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:05,711 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:54:05,711 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x61e72c30 to 127.0.0.1:55020 2023-04-27 03:54:05,711 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:05,711 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:05,711 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:54:05,711 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:05,711 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:54:05,711 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:05,713 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63859c62 to 127.0.0.1:55020 2023-04-27 03:54:05,713 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:05,713 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x56c8e45e to 127.0.0.1:55020 2023-04-27 03:54:05,713 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:05,714 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6038a026 to 127.0.0.1:55020 2023-04-27 03:54:05,714 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:05,714 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-27 03:54:05,714 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 78 2023-04-27 03:54:05,714 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 73 2023-04-27 03:54:05,714 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 144 2023-04-27 03:54:05,714 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 83 2023-04-27 03:54:05,714 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 76 2023-04-27 03:54:05,714 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-27 03:54:05,714 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(398): read 4174 2023-04-27 03:54:05,714 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(398): read 4353 2023-04-27 03:54:05,714 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-27 03:54:05,715 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(402): scanned 1842 2023-04-27 03:54:05,715 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(403): verified 5518 rows 2023-04-27 03:54:05,715 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(402): scanned 1827 2023-04-27 03:54:05,715 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(403): verified 5479 rows 2023-04-27 03:54:05,715 INFO [Listener at localhost/33933] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-27 03:54:05,715 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x44ef43e3 to 127.0.0.1:55020 2023-04-27 03:54:05,715 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:05,724 INFO [Listener at localhost/33933] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-27 03:54:05,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-27 03:54:05,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-27 03:54:05,738 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567645738"}]},"ts":"1682567645738"} 2023-04-27 03:54:05,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-27 03:54:05,739 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-27 03:54:05,820 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-27 03:54:05,822 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2b04f11bbacd80b8750d75eb2d0fbb35, UNASSIGN}] 2023-04-27 03:54:05,824 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2b04f11bbacd80b8750d75eb2d0fbb35, UNASSIGN 2023-04-27 03:54:05,825 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=2b04f11bbacd80b8750d75eb2d0fbb35, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:05,825 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567645825"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682567645825"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682567645825"}]},"ts":"1682567645825"} 2023-04-27 03:54:05,826 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305}] 2023-04-27 03:54:05,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-27 03:54:05,986 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:54:05,986 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2b04f11bbacd80b8750d75eb2d0fbb35, disabling compactions & flushes 2023-04-27 03:54:05,986 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:06,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-27 03:54:06,122 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/feb8d3f4c01a469a9c50ac1f68aaebc4 2023-04-27 03:54:06,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-27 03:54:06,534 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/07d3bb54a0f64184a90370a758b746bf 2023-04-27 03:54:06,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-27 03:54:06,946 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/ceb730290a8742d6bc74511868ef8c2f 2023-04-27 03:54:06,951 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/feb8d3f4c01a469a9c50ac1f68aaebc4 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/feb8d3f4c01a469a9c50ac1f68aaebc4 2023-04-27 03:54:06,955 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/feb8d3f4c01a469a9c50ac1f68aaebc4, entries=150, sequenceid=562, filesize=11.9 K 2023-04-27 03:54:06,956 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/07d3bb54a0f64184a90370a758b746bf as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/07d3bb54a0f64184a90370a758b746bf 2023-04-27 03:54:06,960 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/07d3bb54a0f64184a90370a758b746bf, entries=150, sequenceid=562, filesize=11.9 K 2023-04-27 03:54:06,961 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/ceb730290a8742d6bc74511868ef8c2f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/ceb730290a8742d6bc74511868ef8c2f 2023-04-27 03:54:06,965 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/ceb730290a8742d6bc74511868ef8c2f, entries=150, sequenceid=562, filesize=11.9 K 2023-04-27 03:54:06,966 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=20.13 KB/20610 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 1256ms, sequenceid=562, compaction requested=true 2023-04-27 03:54:06,966 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:54:06,966 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:06,967 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:06,967 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:06,967 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. because compaction request was cancelled 2023-04-27 03:54:06,967 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. after waiting 0 ms 2023-04-27 03:54:06,967 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. because compaction request was cancelled 2023-04-27 03:54:06,967 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:06,967 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:06,967 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. because compaction request was cancelled 2023-04-27 03:54:06,967 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:06,967 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 2b04f11bbacd80b8750d75eb2d0fbb35 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-27 03:54:06,967 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=A 2023-04-27 03:54:06,967 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:06,967 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=B 2023-04-27 03:54:06,967 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:06,967 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2b04f11bbacd80b8750d75eb2d0fbb35, store=C 2023-04-27 03:54:06,968 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:07,378 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/2f61d3bbd7834f68b119bf9014d8f628 2023-04-27 03:54:07,390 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/da3b518cc5a7489aaecdf64d360726d1 2023-04-27 03:54:07,804 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/ab5fd65a04bf4bfea609c1e742870cba 2023-04-27 03:54:07,810 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/A/2f61d3bbd7834f68b119bf9014d8f628 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/2f61d3bbd7834f68b119bf9014d8f628 2023-04-27 03:54:07,815 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/2f61d3bbd7834f68b119bf9014d8f628, entries=150, sequenceid=568, filesize=11.9 K 2023-04-27 03:54:07,817 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/B/da3b518cc5a7489aaecdf64d360726d1 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/da3b518cc5a7489aaecdf64d360726d1 2023-04-27 03:54:07,822 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/da3b518cc5a7489aaecdf64d360726d1, entries=150, sequenceid=568, filesize=11.9 K 2023-04-27 03:54:07,823 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/.tmp/C/ab5fd65a04bf4bfea609c1e742870cba as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/ab5fd65a04bf4bfea609c1e742870cba 2023-04-27 03:54:07,829 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/ab5fd65a04bf4bfea609c1e742870cba, entries=150, sequenceid=568, filesize=11.9 K 2023-04-27 03:54:07,829 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 2b04f11bbacd80b8750d75eb2d0fbb35 in 862ms, sequenceid=568, compaction requested=true 2023-04-27 03:54:07,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-27 03:54:07,853 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/290dc2b81b82472a93ec481e035ede24, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/07af5d9ed7e645929a99a4b91727aad3, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/9210223583d54ca89ffc748d6318154f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/4d9e7354066f4d9ebae58760ffc9dc9f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/fc259a044e954d0bb8febb6e76c741e9, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/8fb41cd0c8584d969865e91211672c6b, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/e35a73efddf54835936ca56e544ec084, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/4c437e0b86e94225b34e839c9976866a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/520cd0056c744611b6afe8a028c811bf, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/62244adb83cb442c9fd3c8d5f6af0774, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/1b0095364f4048eaa578abd6422275c8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/1f9915701d0348568310f503bb4cb204, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/92d8f0f012504a9ab9bd1b2778ee168e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/2b064e1fef164267a9fe4bf5e76485c2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/06d224ab17ac43a0b882f9588895bd86, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/95a843b25ff2431a84bff89c2dbf59be, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/152c07b5916844fc9233f68601fc337e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/cfde396119634d3ca179bb1abca8db21, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/9e9266c4a08147b7834a3a0910518bb8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/f56179240e5749f29f78628dc5fedff4, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/b4312b8e1f904a1b9ff7cfe501e5d976, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/8080140782a044cd880cac811a541f35, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/112c6f6dab5c438892f42025c696582e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/543aa6d428d043e1863d36d2e71cb9a1, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/1929a49af4d0450183b1a5907a45c23a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/3805a27beb40460c8f41a76d0384b78a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/da32a867ba3c4ebf87b875e0fbc2d024, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/1ae831fd16c44b65bdac19667c5a28ab, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/e0778c8917e343c9a51b06e8c12d26ef, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/283c1f28e6104b29a56d916a56a725b3, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/fabcd20dbdac4ebc88ad24b9aa03f1f4, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/88c7df9e43d14eae97f20c13cc23b791, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/f457f9983753480b89a8c64fae5c10d5] to archive 2023-04-27 03:54:07,855 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 03:54:07,861 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/290dc2b81b82472a93ec481e035ede24 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/290dc2b81b82472a93ec481e035ede24 2023-04-27 03:54:07,863 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/07af5d9ed7e645929a99a4b91727aad3 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/07af5d9ed7e645929a99a4b91727aad3 2023-04-27 03:54:07,864 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/9210223583d54ca89ffc748d6318154f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/9210223583d54ca89ffc748d6318154f 2023-04-27 03:54:07,865 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/4d9e7354066f4d9ebae58760ffc9dc9f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/4d9e7354066f4d9ebae58760ffc9dc9f 2023-04-27 03:54:07,867 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/fc259a044e954d0bb8febb6e76c741e9 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/fc259a044e954d0bb8febb6e76c741e9 2023-04-27 03:54:07,869 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/8fb41cd0c8584d969865e91211672c6b to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/8fb41cd0c8584d969865e91211672c6b 2023-04-27 03:54:07,870 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/e35a73efddf54835936ca56e544ec084 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/e35a73efddf54835936ca56e544ec084 2023-04-27 03:54:07,872 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/4c437e0b86e94225b34e839c9976866a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/4c437e0b86e94225b34e839c9976866a 2023-04-27 03:54:07,873 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/520cd0056c744611b6afe8a028c811bf to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/520cd0056c744611b6afe8a028c811bf 2023-04-27 03:54:07,874 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/62244adb83cb442c9fd3c8d5f6af0774 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/62244adb83cb442c9fd3c8d5f6af0774 2023-04-27 03:54:07,875 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/1b0095364f4048eaa578abd6422275c8 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/1b0095364f4048eaa578abd6422275c8 2023-04-27 03:54:07,877 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/1f9915701d0348568310f503bb4cb204 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/1f9915701d0348568310f503bb4cb204 2023-04-27 03:54:07,878 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/92d8f0f012504a9ab9bd1b2778ee168e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/92d8f0f012504a9ab9bd1b2778ee168e 2023-04-27 03:54:07,879 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/2b064e1fef164267a9fe4bf5e76485c2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/2b064e1fef164267a9fe4bf5e76485c2 2023-04-27 03:54:07,880 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/06d224ab17ac43a0b882f9588895bd86 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/06d224ab17ac43a0b882f9588895bd86 2023-04-27 03:54:07,882 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/95a843b25ff2431a84bff89c2dbf59be to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/95a843b25ff2431a84bff89c2dbf59be 2023-04-27 03:54:07,883 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/152c07b5916844fc9233f68601fc337e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/152c07b5916844fc9233f68601fc337e 2023-04-27 03:54:07,884 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/cfde396119634d3ca179bb1abca8db21 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/cfde396119634d3ca179bb1abca8db21 2023-04-27 03:54:07,885 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/9e9266c4a08147b7834a3a0910518bb8 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/9e9266c4a08147b7834a3a0910518bb8 2023-04-27 03:54:07,886 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/f56179240e5749f29f78628dc5fedff4 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/f56179240e5749f29f78628dc5fedff4 2023-04-27 03:54:07,887 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/b4312b8e1f904a1b9ff7cfe501e5d976 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/b4312b8e1f904a1b9ff7cfe501e5d976 2023-04-27 03:54:07,889 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/8080140782a044cd880cac811a541f35 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/8080140782a044cd880cac811a541f35 2023-04-27 03:54:07,890 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/112c6f6dab5c438892f42025c696582e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/112c6f6dab5c438892f42025c696582e 2023-04-27 03:54:07,891 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/543aa6d428d043e1863d36d2e71cb9a1 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/543aa6d428d043e1863d36d2e71cb9a1 2023-04-27 03:54:07,892 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/1929a49af4d0450183b1a5907a45c23a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/1929a49af4d0450183b1a5907a45c23a 2023-04-27 03:54:07,893 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/3805a27beb40460c8f41a76d0384b78a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/3805a27beb40460c8f41a76d0384b78a 2023-04-27 03:54:07,894 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/da32a867ba3c4ebf87b875e0fbc2d024 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/da32a867ba3c4ebf87b875e0fbc2d024 2023-04-27 03:54:07,895 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/1ae831fd16c44b65bdac19667c5a28ab to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/1ae831fd16c44b65bdac19667c5a28ab 2023-04-27 03:54:07,897 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/e0778c8917e343c9a51b06e8c12d26ef to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/e0778c8917e343c9a51b06e8c12d26ef 2023-04-27 03:54:07,898 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/283c1f28e6104b29a56d916a56a725b3 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/283c1f28e6104b29a56d916a56a725b3 2023-04-27 03:54:07,899 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/fabcd20dbdac4ebc88ad24b9aa03f1f4 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/fabcd20dbdac4ebc88ad24b9aa03f1f4 2023-04-27 03:54:07,900 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/88c7df9e43d14eae97f20c13cc23b791 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/88c7df9e43d14eae97f20c13cc23b791 2023-04-27 03:54:07,902 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/f457f9983753480b89a8c64fae5c10d5 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/f457f9983753480b89a8c64fae5c10d5 2023-04-27 03:54:07,925 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/19ea049a37b04fc9ad421c77fe06e7e1, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/81d5388308124386bef5b4a92941edc2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/aef355e335fa46e099cde443f2018f48, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/10895b2bf00e43c4bc136a8dcfe373f2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/df00308aa3d14f859efca7649e969b71, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/9c90a21c05664d53b1c4d23f3a4694d7, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/a6fc24ae4b544892a9c82fcd498ca4f5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/2861b217e13a4a9688a18b70d919c321, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/862a687c448f4ba1b8345402a92481bb, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/00fb19af768f46ec92650aeb523b559f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/2d738780e9664a37a2a057454002a36b, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/81a647d6d2a047d1ba63790e2c090064, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/3d67f32ca8f94546abad605f17dc4f77, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/ad5abd13e84e498082c7f169a7bb07d7, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/491136f8d77f4563a052cf23a6393a8a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/45d82f3b8baf4b5c98678f7beb633e97, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/a7f44a67a6ed45e5835a6f5573a351b8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/5c8817e801a347b580d67b2ef502e8cd, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/ace675ba25b840d3bce9d5a524be25fe, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/e5317f30301b46349a7c3f2d21857919, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/7f8bafed60a14718bd366d4b528b4894, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/f9b0a1f63a5b402a981a2436aa0efcff, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/06c006c3168d44b485137cf9e63579d8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/628e24b6f9e24bf8bd3d4cfaa28c4d8a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/f7296eef8d0c4e3095044897ad3b9baa, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/c2e4ced10deb40c195c58e7d7ddbe05f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/cf3a2c1184e84e33b572c2d139906f25, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/9c248c7cd426479b914121e159b95380, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/b861edb5cd7545f0bfb4e2bf7eb9114a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/aea24cb3ae7540fd9a2f74ce8d2ade10, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/4c2319f7090043c9870943ed95593ab3, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/8cf19a383ff1439490d986086ab3b6c5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/b2505b26be454ce0ab2f17374a520126] to archive 2023-04-27 03:54:07,926 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 03:54:07,927 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/19ea049a37b04fc9ad421c77fe06e7e1 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/19ea049a37b04fc9ad421c77fe06e7e1 2023-04-27 03:54:07,929 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/81d5388308124386bef5b4a92941edc2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/81d5388308124386bef5b4a92941edc2 2023-04-27 03:54:07,930 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/aef355e335fa46e099cde443f2018f48 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/aef355e335fa46e099cde443f2018f48 2023-04-27 03:54:07,931 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/10895b2bf00e43c4bc136a8dcfe373f2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/10895b2bf00e43c4bc136a8dcfe373f2 2023-04-27 03:54:07,932 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/df00308aa3d14f859efca7649e969b71 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/df00308aa3d14f859efca7649e969b71 2023-04-27 03:54:07,933 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/9c90a21c05664d53b1c4d23f3a4694d7 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/9c90a21c05664d53b1c4d23f3a4694d7 2023-04-27 03:54:07,934 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/a6fc24ae4b544892a9c82fcd498ca4f5 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/a6fc24ae4b544892a9c82fcd498ca4f5 2023-04-27 03:54:07,935 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/2861b217e13a4a9688a18b70d919c321 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/2861b217e13a4a9688a18b70d919c321 2023-04-27 03:54:07,936 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/862a687c448f4ba1b8345402a92481bb to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/862a687c448f4ba1b8345402a92481bb 2023-04-27 03:54:07,937 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/00fb19af768f46ec92650aeb523b559f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/00fb19af768f46ec92650aeb523b559f 2023-04-27 03:54:07,938 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/2d738780e9664a37a2a057454002a36b to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/2d738780e9664a37a2a057454002a36b 2023-04-27 03:54:07,939 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/81a647d6d2a047d1ba63790e2c090064 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/81a647d6d2a047d1ba63790e2c090064 2023-04-27 03:54:07,940 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/3d67f32ca8f94546abad605f17dc4f77 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/3d67f32ca8f94546abad605f17dc4f77 2023-04-27 03:54:07,941 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/ad5abd13e84e498082c7f169a7bb07d7 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/ad5abd13e84e498082c7f169a7bb07d7 2023-04-27 03:54:07,942 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/491136f8d77f4563a052cf23a6393a8a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/491136f8d77f4563a052cf23a6393a8a 2023-04-27 03:54:07,943 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/45d82f3b8baf4b5c98678f7beb633e97 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/45d82f3b8baf4b5c98678f7beb633e97 2023-04-27 03:54:07,944 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/a7f44a67a6ed45e5835a6f5573a351b8 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/a7f44a67a6ed45e5835a6f5573a351b8 2023-04-27 03:54:07,945 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/5c8817e801a347b580d67b2ef502e8cd to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/5c8817e801a347b580d67b2ef502e8cd 2023-04-27 03:54:07,946 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/ace675ba25b840d3bce9d5a524be25fe to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/ace675ba25b840d3bce9d5a524be25fe 2023-04-27 03:54:07,947 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/e5317f30301b46349a7c3f2d21857919 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/e5317f30301b46349a7c3f2d21857919 2023-04-27 03:54:07,948 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/7f8bafed60a14718bd366d4b528b4894 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/7f8bafed60a14718bd366d4b528b4894 2023-04-27 03:54:07,949 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/f9b0a1f63a5b402a981a2436aa0efcff to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/f9b0a1f63a5b402a981a2436aa0efcff 2023-04-27 03:54:07,950 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/06c006c3168d44b485137cf9e63579d8 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/06c006c3168d44b485137cf9e63579d8 2023-04-27 03:54:07,951 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/628e24b6f9e24bf8bd3d4cfaa28c4d8a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/628e24b6f9e24bf8bd3d4cfaa28c4d8a 2023-04-27 03:54:07,952 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/f7296eef8d0c4e3095044897ad3b9baa to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/f7296eef8d0c4e3095044897ad3b9baa 2023-04-27 03:54:07,953 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/c2e4ced10deb40c195c58e7d7ddbe05f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/c2e4ced10deb40c195c58e7d7ddbe05f 2023-04-27 03:54:07,954 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/cf3a2c1184e84e33b572c2d139906f25 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/cf3a2c1184e84e33b572c2d139906f25 2023-04-27 03:54:07,955 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/9c248c7cd426479b914121e159b95380 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/9c248c7cd426479b914121e159b95380 2023-04-27 03:54:07,957 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/b861edb5cd7545f0bfb4e2bf7eb9114a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/b861edb5cd7545f0bfb4e2bf7eb9114a 2023-04-27 03:54:07,958 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/aea24cb3ae7540fd9a2f74ce8d2ade10 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/aea24cb3ae7540fd9a2f74ce8d2ade10 2023-04-27 03:54:07,959 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/4c2319f7090043c9870943ed95593ab3 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/4c2319f7090043c9870943ed95593ab3 2023-04-27 03:54:07,960 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/8cf19a383ff1439490d986086ab3b6c5 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/8cf19a383ff1439490d986086ab3b6c5 2023-04-27 03:54:07,961 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/b2505b26be454ce0ab2f17374a520126 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/b2505b26be454ce0ab2f17374a520126 2023-04-27 03:54:07,964 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/01f0e954bd904a248996e50d5d9ede70, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/60616a58a61943a1a26872c232066fe7, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/92a04ba2acdb4c28a7e7824773590036, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/1c1955c6a031432f9b30827f89e3d060, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/00c3f9aab5ad4126be46c240e93d4eb3, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/089e53dc16e54371b6a716f071b85efe, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/523c68810fea48a280fd571c0cfe79ce, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/16609fe422bc44379a15b149b59487ec, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/1844e6156bce4dd995ea97da046f6bcd, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/9d8c985afa814970be62721548859585, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/3466f481f3844a8eb4eb68e4325e65f4, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/d07450635f734fffb96287257b67985b, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/7470ae35249c4594867dfdba4ba1a4a5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/90f3ed99ac784aa18ec3d95ed16a22ed, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/eb37be95781349ceaf05d20f61f4b2eb, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/6aefb67646fb488ca273132a6e53d174, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/6e476e4656bf4d9697923e4f56073113, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/8a9b47b7bfdf458594951f04aef3029d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/7facb8e0cf284125af54a07b20af81e4, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/81c2e0edd58b493cb31a31e653615a2f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/8714328911c64eb58e76e84361a83caf, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/b2940039c8e645b2b3686905d2c7b596, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/6be0734d02c54fe4aa7a1ec89bed28de, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/79b4152c94d54c0eb69d65f6d01600f6, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/130431b6f9124c44bdb0a0c69f830204, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/603695e6d54444c09a4c06e1b2d4b63c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/a4ee3583d0ba4a88b0b91e3d1e061238, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/5882ea23cef7451f9ce867e233578fdd, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/9a3f126351654b4a94f362645a0b311d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/213b218d714d45c2b71e8e577631be7c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/dad49b6745ab416b8a05e7d2ae5f7871, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/19c53a06a75a43fe8fe580b69d2686c2] to archive 2023-04-27 03:54:07,965 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 03:54:07,966 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/01f0e954bd904a248996e50d5d9ede70 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/01f0e954bd904a248996e50d5d9ede70 2023-04-27 03:54:07,967 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/60616a58a61943a1a26872c232066fe7 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/60616a58a61943a1a26872c232066fe7 2023-04-27 03:54:07,968 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/92a04ba2acdb4c28a7e7824773590036 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/92a04ba2acdb4c28a7e7824773590036 2023-04-27 03:54:07,969 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/1c1955c6a031432f9b30827f89e3d060 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/1c1955c6a031432f9b30827f89e3d060 2023-04-27 03:54:07,970 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/00c3f9aab5ad4126be46c240e93d4eb3 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/00c3f9aab5ad4126be46c240e93d4eb3 2023-04-27 03:54:07,971 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/089e53dc16e54371b6a716f071b85efe to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/089e53dc16e54371b6a716f071b85efe 2023-04-27 03:54:07,972 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/523c68810fea48a280fd571c0cfe79ce to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/523c68810fea48a280fd571c0cfe79ce 2023-04-27 03:54:07,974 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/16609fe422bc44379a15b149b59487ec to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/16609fe422bc44379a15b149b59487ec 2023-04-27 03:54:07,975 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/1844e6156bce4dd995ea97da046f6bcd to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/1844e6156bce4dd995ea97da046f6bcd 2023-04-27 03:54:07,977 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/9d8c985afa814970be62721548859585 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/9d8c985afa814970be62721548859585 2023-04-27 03:54:07,978 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/3466f481f3844a8eb4eb68e4325e65f4 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/3466f481f3844a8eb4eb68e4325e65f4 2023-04-27 03:54:07,979 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/d07450635f734fffb96287257b67985b to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/d07450635f734fffb96287257b67985b 2023-04-27 03:54:07,980 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/7470ae35249c4594867dfdba4ba1a4a5 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/7470ae35249c4594867dfdba4ba1a4a5 2023-04-27 03:54:07,982 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/90f3ed99ac784aa18ec3d95ed16a22ed to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/90f3ed99ac784aa18ec3d95ed16a22ed 2023-04-27 03:54:07,983 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/eb37be95781349ceaf05d20f61f4b2eb to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/eb37be95781349ceaf05d20f61f4b2eb 2023-04-27 03:54:07,984 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/6aefb67646fb488ca273132a6e53d174 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/6aefb67646fb488ca273132a6e53d174 2023-04-27 03:54:07,986 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/6e476e4656bf4d9697923e4f56073113 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/6e476e4656bf4d9697923e4f56073113 2023-04-27 03:54:07,987 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/8a9b47b7bfdf458594951f04aef3029d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/8a9b47b7bfdf458594951f04aef3029d 2023-04-27 03:54:07,989 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/7facb8e0cf284125af54a07b20af81e4 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/7facb8e0cf284125af54a07b20af81e4 2023-04-27 03:54:07,992 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/81c2e0edd58b493cb31a31e653615a2f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/81c2e0edd58b493cb31a31e653615a2f 2023-04-27 03:54:07,993 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/8714328911c64eb58e76e84361a83caf to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/8714328911c64eb58e76e84361a83caf 2023-04-27 03:54:07,995 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/b2940039c8e645b2b3686905d2c7b596 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/b2940039c8e645b2b3686905d2c7b596 2023-04-27 03:54:07,996 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/6be0734d02c54fe4aa7a1ec89bed28de to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/6be0734d02c54fe4aa7a1ec89bed28de 2023-04-27 03:54:07,997 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/79b4152c94d54c0eb69d65f6d01600f6 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/79b4152c94d54c0eb69d65f6d01600f6 2023-04-27 03:54:07,998 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/130431b6f9124c44bdb0a0c69f830204 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/130431b6f9124c44bdb0a0c69f830204 2023-04-27 03:54:07,999 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/603695e6d54444c09a4c06e1b2d4b63c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/603695e6d54444c09a4c06e1b2d4b63c 2023-04-27 03:54:08,001 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/a4ee3583d0ba4a88b0b91e3d1e061238 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/a4ee3583d0ba4a88b0b91e3d1e061238 2023-04-27 03:54:08,002 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/5882ea23cef7451f9ce867e233578fdd to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/5882ea23cef7451f9ce867e233578fdd 2023-04-27 03:54:08,004 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/9a3f126351654b4a94f362645a0b311d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/9a3f126351654b4a94f362645a0b311d 2023-04-27 03:54:08,005 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/213b218d714d45c2b71e8e577631be7c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/213b218d714d45c2b71e8e577631be7c 2023-04-27 03:54:08,007 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/dad49b6745ab416b8a05e7d2ae5f7871 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/dad49b6745ab416b8a05e7d2ae5f7871 2023-04-27 03:54:08,009 DEBUG [StoreCloser-TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/19c53a06a75a43fe8fe580b69d2686c2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/19c53a06a75a43fe8fe580b69d2686c2 2023-04-27 03:54:08,015 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/recovered.edits/571.seqid, newMaxSeqId=571, maxSeqId=1 2023-04-27 03:54:08,017 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35. 2023-04-27 03:54:08,017 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2b04f11bbacd80b8750d75eb2d0fbb35: 2023-04-27 03:54:08,018 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:54:08,019 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=2b04f11bbacd80b8750d75eb2d0fbb35, regionState=CLOSED 2023-04-27 03:54:08,019 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567648019"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682567648019"}]},"ts":"1682567648019"} 2023-04-27 03:54:08,022 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-27 03:54:08,022 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 2b04f11bbacd80b8750d75eb2d0fbb35, server=jenkins-hbase4.apache.org,40885,1682567618305 in 2.1940 sec 2023-04-27 03:54:08,024 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-27 03:54:08,024 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2b04f11bbacd80b8750d75eb2d0fbb35, UNASSIGN in 2.2000 sec 2023-04-27 03:54:08,025 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567648025"}]},"ts":"1682567648025"} 2023-04-27 03:54:08,026 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-27 03:54:08,027 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-27 03:54:08,029 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.2980 sec 2023-04-27 03:54:09,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-27 03:54:09,847 INFO [Listener at localhost/33933] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-27 03:54:09,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-27 03:54:09,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:54:09,860 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:54:09,862 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:54:09,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-27 03:54:09,867 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:54:09,877 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A, FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B, FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C, FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/recovered.edits] 2023-04-27 03:54:09,881 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/2f61d3bbd7834f68b119bf9014d8f628 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/2f61d3bbd7834f68b119bf9014d8f628 2023-04-27 03:54:09,884 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/6515501824c643a2b3f45f34908a895c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/6515501824c643a2b3f45f34908a895c 2023-04-27 03:54:09,885 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/a83217b03c34442c922c2994d71440ac to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/a83217b03c34442c922c2994d71440ac 2023-04-27 03:54:09,887 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/feb8d3f4c01a469a9c50ac1f68aaebc4 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/feb8d3f4c01a469a9c50ac1f68aaebc4 2023-04-27 03:54:09,889 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/feccbc6f3c134a6587a74e53afff3d32 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/A/feccbc6f3c134a6587a74e53afff3d32 2023-04-27 03:54:09,892 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/07d3bb54a0f64184a90370a758b746bf to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/07d3bb54a0f64184a90370a758b746bf 2023-04-27 03:54:09,893 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/2b40773950314450b492f792e17700dc to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/2b40773950314450b492f792e17700dc 2023-04-27 03:54:09,895 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/4762cebad68b48d7bf262bd61b7e7ab5 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/4762cebad68b48d7bf262bd61b7e7ab5 2023-04-27 03:54:09,897 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/7ec0137e100c47f9a01988d00d7a3959 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/7ec0137e100c47f9a01988d00d7a3959 2023-04-27 03:54:09,898 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/da3b518cc5a7489aaecdf64d360726d1 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/B/da3b518cc5a7489aaecdf64d360726d1 2023-04-27 03:54:09,901 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/148c63bb1cfa4a7cb11e73e8213dc394 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/148c63bb1cfa4a7cb11e73e8213dc394 2023-04-27 03:54:09,902 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/5a68a0a2239b400db5258f7dc9bbbeda to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/5a68a0a2239b400db5258f7dc9bbbeda 2023-04-27 03:54:09,903 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/ab5fd65a04bf4bfea609c1e742870cba to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/ab5fd65a04bf4bfea609c1e742870cba 2023-04-27 03:54:09,904 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/ceb730290a8742d6bc74511868ef8c2f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/ceb730290a8742d6bc74511868ef8c2f 2023-04-27 03:54:09,905 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/fe04b330d3e245dda8889fb4db548538 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/C/fe04b330d3e245dda8889fb4db548538 2023-04-27 03:54:09,908 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/recovered.edits/571.seqid to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35/recovered.edits/571.seqid 2023-04-27 03:54:09,909 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/2b04f11bbacd80b8750d75eb2d0fbb35 2023-04-27 03:54:09,909 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-27 03:54:09,912 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:54:09,932 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-27 03:54:09,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-27 03:54:09,974 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-27 03:54:09,976 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:54:09,976 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-27 03:54:09,976 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682567649976"}]},"ts":"9223372036854775807"} 2023-04-27 03:54:09,978 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-27 03:54:09,978 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2b04f11bbacd80b8750d75eb2d0fbb35, NAME => 'TestAcidGuarantees,,1682567622361.2b04f11bbacd80b8750d75eb2d0fbb35.', STARTKEY => '', ENDKEY => ''}] 2023-04-27 03:54:09,978 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-27 03:54:09,979 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682567649978"}]},"ts":"9223372036854775807"} 2023-04-27 03:54:09,980 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-27 03:54:09,982 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:54:09,983 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 130 msec 2023-04-27 03:54:10,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-27 03:54:10,165 INFO [Listener at localhost/33933] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-27 03:54:10,183 INFO [Listener at localhost/33933] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=235 (was 207) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6dfca63e-shared-pool-20 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:40885-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-689348034_15 at /127.0.0.1:60900 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6dfca63e-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6dfca63e-shared-pool-19 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6dfca63e-shared-pool-21 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-689348034_15 at /127.0.0.1:60828 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=569 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=421 (was 299) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=12218 (was 12840) 2023-04-27 03:54:10,196 INFO [Listener at localhost/33933] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=235, OpenFileDescriptor=569, MaxFileDescriptor=60000, SystemLoadAverage=421, ProcessCount=170, AvailableMemoryMB=12216 2023-04-27 03:54:10,199 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-27 03:54:10,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-27 03:54:10,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-27 03:54:10,202 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-27 03:54:10,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-27 03:54:10,203 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-27 03:54:10,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-27 03:54:10,205 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:10,206 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3 empty. 2023-04-27 03:54:10,206 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:10,207 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-27 03:54:10,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-27 03:54:10,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-27 03:54:10,621 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-27 03:54:10,623 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => d66a88af1ba39e44423282d0b046d5b3, NAME => 'TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp 2023-04-27 03:54:10,642 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 03:54:10,642 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing d66a88af1ba39e44423282d0b046d5b3, disabling compactions & flushes 2023-04-27 03:54:10,642 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:10,642 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:10,642 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. after waiting 0 ms 2023-04-27 03:54:10,643 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:10,643 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:10,643 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:10,655 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-27 03:54:10,659 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567650659"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682567650659"}]},"ts":"1682567650659"} 2023-04-27 03:54:10,661 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-27 03:54:10,664 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-27 03:54:10,664 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567650664"}]},"ts":"1682567650664"} 2023-04-27 03:54:10,666 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-27 03:54:10,671 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d66a88af1ba39e44423282d0b046d5b3, ASSIGN}] 2023-04-27 03:54:10,674 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d66a88af1ba39e44423282d0b046d5b3, ASSIGN 2023-04-27 03:54:10,675 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=d66a88af1ba39e44423282d0b046d5b3, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40885,1682567618305; forceNewPlan=false, retain=false 2023-04-27 03:54:10,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-27 03:54:10,826 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=d66a88af1ba39e44423282d0b046d5b3, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:10,826 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567650826"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682567650826"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682567650826"}]},"ts":"1682567650826"} 2023-04-27 03:54:10,828 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305}] 2023-04-27 03:54:10,985 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:10,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d66a88af1ba39e44423282d0b046d5b3, NAME => 'TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.', STARTKEY => '', ENDKEY => ''} 2023-04-27 03:54:10,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:10,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 03:54:10,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:10,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:10,987 INFO [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:10,988 DEBUG [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A 2023-04-27 03:54:10,988 DEBUG [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A 2023-04-27 03:54:10,988 INFO [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:54:10,989 INFO [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d66a88af1ba39e44423282d0b046d5b3 columnFamilyName A 2023-04-27 03:54:10,989 INFO [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] regionserver.HStore(310): Store=d66a88af1ba39e44423282d0b046d5b3/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:54:10,989 INFO [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:10,990 DEBUG [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B 2023-04-27 03:54:10,990 DEBUG [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B 2023-04-27 03:54:10,990 INFO [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:54:10,991 INFO [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d66a88af1ba39e44423282d0b046d5b3 columnFamilyName B 2023-04-27 03:54:10,991 INFO [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] regionserver.HStore(310): Store=d66a88af1ba39e44423282d0b046d5b3/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:54:10,991 INFO [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:10,992 DEBUG [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C 2023-04-27 03:54:10,992 DEBUG [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C 2023-04-27 03:54:10,992 INFO [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:54:10,993 INFO [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d66a88af1ba39e44423282d0b046d5b3 columnFamilyName C 2023-04-27 03:54:10,993 INFO [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] regionserver.HStore(310): Store=d66a88af1ba39e44423282d0b046d5b3/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:54:10,993 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:10,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:10,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:10,996 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-27 03:54:10,997 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:10,999 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-27 03:54:10,999 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d66a88af1ba39e44423282d0b046d5b3; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66930194, jitterRate=-0.0026623904705047607}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-27 03:54:10,999 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:11,000 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., pid=18, masterSystemTime=1682567650981 2023-04-27 03:54:11,001 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:11,001 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:11,002 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=d66a88af1ba39e44423282d0b046d5b3, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:11,002 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567651002"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682567651002"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682567651002"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682567651002"}]},"ts":"1682567651002"} 2023-04-27 03:54:11,005 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-27 03:54:11,005 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 in 175 msec 2023-04-27 03:54:11,007 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-27 03:54:11,007 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d66a88af1ba39e44423282d0b046d5b3, ASSIGN in 334 msec 2023-04-27 03:54:11,007 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-27 03:54:11,008 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567651008"}]},"ts":"1682567651008"} 2023-04-27 03:54:11,009 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-27 03:54:11,011 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-27 03:54:11,013 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 812 msec 2023-04-27 03:54:11,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-27 03:54:11,309 INFO [Listener at localhost/33933] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-27 03:54:11,310 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x5b8d0eb2 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:54:11,315 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52274650, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:54:11,317 DEBUG [Listener at localhost/33933] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:54:11,322 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52120, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:54:11,324 DEBUG [Listener at localhost/33933] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-27 03:54:11,327 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46096, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-27 03:54:11,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-27 03:54:11,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-27 03:54:11,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-27 03:54:11,769 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-27 03:54:11,770 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-27 03:54:11,770 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-27 03:54:11,775 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-27 03:54:11,789 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d66a88af1ba39e44423282d0b046d5b3, REOPEN/MOVE}] 2023-04-27 03:54:11,790 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d66a88af1ba39e44423282d0b046d5b3, REOPEN/MOVE 2023-04-27 03:54:11,791 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=d66a88af1ba39e44423282d0b046d5b3, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:11,791 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567651791"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682567651791"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682567651791"}]},"ts":"1682567651791"} 2023-04-27 03:54:11,792 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305}] 2023-04-27 03:54:11,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:11,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d66a88af1ba39e44423282d0b046d5b3, disabling compactions & flushes 2023-04-27 03:54:11,946 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:11,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:11,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. after waiting 0 ms 2023-04-27 03:54:11,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:11,951 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-27 03:54:11,952 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:11,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:11,952 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: d66a88af1ba39e44423282d0b046d5b3 to self. 2023-04-27 03:54:11,954 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:11,954 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=d66a88af1ba39e44423282d0b046d5b3, regionState=CLOSED 2023-04-27 03:54:11,955 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567651954"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682567651954"}]},"ts":"1682567651954"} 2023-04-27 03:54:11,958 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-27 03:54:11,958 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 in 164 msec 2023-04-27 03:54:11,959 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=d66a88af1ba39e44423282d0b046d5b3, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,40885,1682567618305; forceNewPlan=false, retain=true 2023-04-27 03:54:12,109 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=d66a88af1ba39e44423282d0b046d5b3, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,110 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567652109"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682567652109"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682567652109"}]},"ts":"1682567652109"} 2023-04-27 03:54:12,113 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305}] 2023-04-27 03:54:12,268 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:12,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d66a88af1ba39e44423282d0b046d5b3, NAME => 'TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.', STARTKEY => '', ENDKEY => ''} 2023-04-27 03:54:12,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:12,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 03:54:12,269 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:12,269 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:12,271 INFO [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:12,272 DEBUG [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A 2023-04-27 03:54:12,272 DEBUG [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A 2023-04-27 03:54:12,272 INFO [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:54:12,275 INFO [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d66a88af1ba39e44423282d0b046d5b3 columnFamilyName A 2023-04-27 03:54:12,277 INFO [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] regionserver.HStore(310): Store=d66a88af1ba39e44423282d0b046d5b3/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:54:12,277 INFO [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:12,278 DEBUG [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B 2023-04-27 03:54:12,278 DEBUG [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B 2023-04-27 03:54:12,278 INFO [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:54:12,279 INFO [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d66a88af1ba39e44423282d0b046d5b3 columnFamilyName B 2023-04-27 03:54:12,279 INFO [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] regionserver.HStore(310): Store=d66a88af1ba39e44423282d0b046d5b3/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:54:12,280 INFO [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:12,280 DEBUG [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C 2023-04-27 03:54:12,280 DEBUG [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C 2023-04-27 03:54:12,280 INFO [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:54:12,281 INFO [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d66a88af1ba39e44423282d0b046d5b3 columnFamilyName C 2023-04-27 03:54:12,282 INFO [StoreOpener-d66a88af1ba39e44423282d0b046d5b3-1] regionserver.HStore(310): Store=d66a88af1ba39e44423282d0b046d5b3/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:54:12,282 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:12,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:12,284 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:12,285 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-27 03:54:12,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:12,288 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d66a88af1ba39e44423282d0b046d5b3; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73556367, jitterRate=0.09607528150081635}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-27 03:54:12,288 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:12,288 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., pid=23, masterSystemTime=1682567652265 2023-04-27 03:54:12,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:12,290 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:12,291 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=d66a88af1ba39e44423282d0b046d5b3, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,291 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567652290"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682567652290"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682567652290"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682567652290"}]},"ts":"1682567652290"} 2023-04-27 03:54:12,294 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-27 03:54:12,294 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 in 179 msec 2023-04-27 03:54:12,296 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-27 03:54:12,296 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d66a88af1ba39e44423282d0b046d5b3, REOPEN/MOVE in 505 msec 2023-04-27 03:54:12,299 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-27 03:54:12,299 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 524 msec 2023-04-27 03:54:12,301 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 960 msec 2023-04-27 03:54:12,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-27 03:54:12,311 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x051c62b6 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:54:12,316 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21281a37, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:54:12,317 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x0a71edf0 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:54:12,321 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2bab6b2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:54:12,322 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x21265fcb to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:54:12,327 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@24bd3872, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:54:12,327 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x7aba8e46 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:54:12,332 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@370947cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:54:12,333 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x0feb770d to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:54:12,338 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@210d1008, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:54:12,338 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x00c2598b to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:54:12,343 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@498694cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:54:12,343 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x508246cc to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:54:12,349 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@421e5356, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:54:12,349 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x08795451 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:54:12,354 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a0fd7b4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:54:12,354 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x58e27a32 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:54:12,358 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a7b66a3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:54:12,370 DEBUG [hconnection-0x76152026-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:54:12,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:12,373 DEBUG [hconnection-0x2d14f273-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:54:12,373 DEBUG [hconnection-0x2ce3ee4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:54:12,375 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52128, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:54:12,380 DEBUG [hconnection-0x5114f190-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:54:12,382 DEBUG [hconnection-0xbf0e4ef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:54:12,382 DEBUG [hconnection-0x21fbf1ab-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:54:12,385 DEBUG [hconnection-0x15018fae-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:54:12,385 DEBUG [hconnection-0x11c7dd67-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:54:12,385 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52150, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:54:12,385 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52154, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:54:12,386 DEBUG [hconnection-0x5d7da072-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:54:12,386 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52140, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:54:12,386 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52164, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:54:12,386 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52168, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:54:12,386 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52160, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:54:12,393 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52176, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:54:12,393 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52180, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:54:12,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:12,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:12,400 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:12,400 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:12,400 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:12,400 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:12,401 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,401 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:12,402 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:12,402 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:12,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:12,403 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,403 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:12,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:12,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:12,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:12,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:12,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:12,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:12,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:12,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:12,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:12,412 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:12,412 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:12,412 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:12,412 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:12,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:12,415 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:12,415 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d66a88af1ba39e44423282d0b046d5b3 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:54:12,416 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:12,417 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:12,417 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:12,417 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,417 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:12,417 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:12,420 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=A 2023-04-27 03:54:12,420 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:12,420 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=B 2023-04-27 03:54:12,420 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:12,420 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=C 2023-04-27 03:54:12,420 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:12,421 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,421 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:12,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:12,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:12,422 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,422 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:12,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:12,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:12,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:12,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fc05d54[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:12,427 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:12,427 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,428 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,428 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:12,428 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:12,428 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:12,430 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,430 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:12,431 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:12,431 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,431 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:12,432 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:12,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:12,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:12,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:12,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:12,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:12,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:12,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:12,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:12,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:12,435 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:12,435 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:12,435 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:12,435 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:12,435 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:12,437 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,437 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:12,437 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:12,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:12,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:12,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:12,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:12,438 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:12,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:12,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,440 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,446 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:12,446 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:12,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:12,446 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:12,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:12,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:12,446 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:12,447 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:12,447 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:12,447 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:12,447 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,447 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,447 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,447 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,447 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:12,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:12,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:12,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567652445, max timestamp=1682567652454Num uniques 859; 2023-04-27 03:54:12,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:54:12,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567652445, max timestamp=1682567652454Num uniques 859; 2023-04-27 03:54:12,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:54:12,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567652445, max timestamp=1682567652454Num uniques 859; 2023-04-27 03:54:12,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:54:12,476 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304272e4101d935ac4bbf9dddf7f69742d4c7 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272e4101d935ac4bbf9dddf7f69742d4c7 2023-04-27 03:54:12,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:12,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567712490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:12,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567712491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:12,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567712524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:12,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567712524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:12,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52150 deadline: 1682567712526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,548 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:12,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:12,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:12,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:12,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:12,564 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:12,564 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:12,564 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:12,564 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:12,565 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,565 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:12,567 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:12,567 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:12,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:12,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:12,568 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,568 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:12,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:12,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:12,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:12,574 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:12,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:12,575 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:12,575 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:12,575 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:12,592 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:12,592 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:12,592 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:12,592 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:12,592 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:12,592 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:12,592 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:12,592 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:12,592 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,596 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:12,596 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:12,598 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:12,599 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:12,599 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:12,600 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:12,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:12,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:12,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:12,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33807b2[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:12,602 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:12,602 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,603 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:12,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:12,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:12,607 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:12,607 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:12,607 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:12,607 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,607 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:12,607 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:12,607 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:12,607 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:12,607 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:12,608 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:12,608 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:12,610 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,610 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,610 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:12,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:12,613 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:12,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:12,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:12,615 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:12,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,619 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:12,619 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:12,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:12,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:12,619 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,619 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:12,619 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:12,619 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,619 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:12,619 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,619 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,619 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:12,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:12,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:12,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:12,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:12,620 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:12,620 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:12,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:12,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567712626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:12,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567712628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:12,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567712641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:12,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567712641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:12,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52150 deadline: 1682567712642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,720 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:12,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:12,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:12,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:12,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:12,727 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:12,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:12,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:12,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:12,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:12,729 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:12,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:12,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:12,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:12,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:12,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:12,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:12,730 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:12,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:12,730 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:12,730 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:12,730 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:12,730 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:12,730 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:12,731 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:12,731 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:12,731 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:12,731 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:12,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:12,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:12,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:12,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:12,733 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,733 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:12,733 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,733 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:12,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:12,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7451a16b[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:12,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:12,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,738 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,738 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:12,742 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,742 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:12,742 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:12,742 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:12,747 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,747 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:12,747 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:12,747 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:12,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:12,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:12,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:12,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:12,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:12,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:12,751 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:12,751 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:12,751 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:12,751 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:12,751 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:12,756 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:12,756 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,756 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:12,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:12,757 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:12,758 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,761 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:12,761 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:12,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:12,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:12,761 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:12,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:12,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:12,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:12,761 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,761 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:12,761 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:12,761 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,762 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,762 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,762 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,762 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:12,762 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:12,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:12,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:12,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567712829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:12,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567712834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:12,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52150 deadline: 1682567712844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:12,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567712845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:12,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567712845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,862 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:12,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:12,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:12,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:12,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:12,875 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:12,875 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:12,875 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:12,875 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:12,876 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,876 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:12,879 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:12,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:12,879 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:12,880 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,880 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:12,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:12,880 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:12,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:12,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:12,881 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:12,881 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:12,881 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:12,888 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:12,888 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:12,888 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:12,888 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:12,888 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:12,888 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:12,889 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:12,889 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:12,889 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,889 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:12,889 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:12,890 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:12,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:12,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:12,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:12,891 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:12,892 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,892 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:12,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:12,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:12,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c22b54[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:12,893 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:12,893 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,894 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,894 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:12,894 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:12,894 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:12,896 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:12,896 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:12,896 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,896 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:12,897 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/799b9c0b636e4d6da0d33ec2e2dda0cb 2023-04-27 03:54:12,897 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:12,897 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:12,897 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:12,897 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:12,897 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:12,897 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:12,897 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:12,900 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:12,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:12,901 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,901 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:12,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:12,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:12,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,903 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:12,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:12,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:12,905 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:12,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:12,908 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,908 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:12,908 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:12,908 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:12,908 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:12,908 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:12,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:12,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:12,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:12,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:12,908 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:12,908 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,908 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:12,908 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:12,908 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:12,908 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:12,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:12,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:12,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:13,009 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:13,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:13,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:13,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:13,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:13,028 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:13,028 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:13,028 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:13,028 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:13,029 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,029 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,030 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:13,030 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:13,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:13,031 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,031 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:13,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:13,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,032 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:13,040 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:13,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:13,040 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:13,041 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:13,041 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:13,041 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:13,041 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:13,042 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:13,042 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:13,042 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:13,042 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:13,042 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:13,042 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:13,042 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,042 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:13,042 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:13,044 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,044 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:13,044 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,044 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:13,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:13,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:13,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:13,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1545ebe5[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:13,046 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:13,046 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,049 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,049 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,049 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:13,049 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:13,050 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:13,051 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:13,052 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,052 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:13,052 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:13,052 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:13,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:13,056 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:13,056 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:13,056 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:13,056 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:13,056 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:13,059 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,060 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:13,060 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:13,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:13,061 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:13,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,062 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,064 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,065 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:13,065 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:13,065 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:13,065 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:13,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:13,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:13,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:13,065 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:13,065 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,065 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,065 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,065 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,065 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:13,065 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:13,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:13,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:13,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:13,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567713135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:13,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567713137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:13,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52150 deadline: 1682567713148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:13,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567713150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:13,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567713150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,165 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:13,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:13,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:13,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:13,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:13,169 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:13,169 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:13,169 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:13,169 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:13,170 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,170 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,172 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,172 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:13,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:13,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:13,173 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,173 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:13,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:13,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:13,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:13,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:13,174 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:13,174 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:13,174 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:13,177 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:13,177 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:13,177 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:13,177 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:13,177 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:13,177 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:13,177 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:13,177 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:13,177 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,177 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:13,177 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:13,179 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,179 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:13,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:13,179 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,179 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:13,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:13,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:13,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:13,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:13,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56837b68[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:13,181 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:13,181 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,182 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,182 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:13,182 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:13,182 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:13,182 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,182 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:13,183 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,183 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:13,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,184 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:13,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:13,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:13,184 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:13,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:13,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:13,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:13,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:13,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:13,186 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:13,186 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:13,186 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:13,187 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:13,187 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:13,189 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,189 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:13,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:13,189 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:13,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:13,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:13,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:13,196 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,197 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:13,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,207 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,207 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:13,207 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:13,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:13,207 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:13,207 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:13,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:13,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:13,207 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,208 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,208 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,208 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,208 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:13,208 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:13,208 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:13,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:13,308 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:13,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:13,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:13,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:13,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:13,313 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:13,313 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:13,313 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:13,313 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:13,313 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,313 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,315 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,315 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:13,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:13,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:13,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,316 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,316 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:13,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:13,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:13,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:13,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:13,317 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:13,317 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:13,317 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:13,318 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:13,318 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:13,318 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:13,318 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:13,318 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:13,318 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:13,318 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:13,318 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:13,318 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,318 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:13,318 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:13,321 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,321 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:13,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:13,321 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,321 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:13,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:13,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:13,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:13,323 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/f437fc8d4fd04f9a931f64fbbf76e4aa 2023-04-27 03:54:13,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:13,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f20c22a[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:13,323 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:13,323 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,324 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,325 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,325 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:13,325 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:13,325 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:13,325 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:13,327 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:13,327 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:13,327 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:13,327 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:13,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:13,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:13,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:13,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:13,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:13,331 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:13,331 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:13,332 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:13,332 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:13,332 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:13,334 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,334 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:13,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,334 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:13,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:13,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:13,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:13,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:13,335 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:13,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,336 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:13,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,341 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,341 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,341 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,341 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:13,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:13,341 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:13,341 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:13,341 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:13,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:13,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:13,341 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:13,341 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,341 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:13,341 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,342 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:13,442 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:13,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:13,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:13,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:13,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:13,446 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:13,446 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:13,446 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:13,446 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:13,446 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,446 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,463 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:13,463 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:13,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:13,464 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,464 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:13,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:13,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:13,465 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:13,465 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:13,465 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:13,465 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:13,465 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:13,465 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:13,465 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:13,465 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:13,465 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:13,466 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:13,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:13,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:13,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:13,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:13,467 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,468 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,468 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:13,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:13,468 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:13,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:13,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:13,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:13,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:13,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@611d411[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:13,470 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:13,470 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,471 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,471 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:13,471 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:13,471 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:13,471 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,471 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:13,473 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:13,473 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:13,473 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,473 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:13,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:13,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:13,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:13,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:13,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:13,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:13,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:13,475 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:13,476 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:13,476 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:13,476 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:13,476 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:13,477 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:13,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:13,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:13,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,478 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,478 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,478 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:13,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:13,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:13,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:13,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,481 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:13,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:13,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:13,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:13,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:13,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:13,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:13,484 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:13,484 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:13,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:13,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:13,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,485 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:13,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:13,584 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-27 03:54:13,584 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:13,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:13,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:13,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:13,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:13,590 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:13,590 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:13,590 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:13,590 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:13,590 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,590 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,592 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:13,592 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:13,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:13,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,592 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,592 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:13,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:13,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:13,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:13,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:13,596 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:13,596 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:13,596 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:13,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:13,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:13,597 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:13,597 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:13,597 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:13,597 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:13,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:13,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:13,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:13,598 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:13,602 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:13,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:13,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:13,603 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,603 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:13,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:13,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:13,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:13,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c311df5[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:13,608 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:13,608 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,610 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,610 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,610 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:13,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:13,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:13,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:13,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,612 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:13,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,612 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:13,613 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:13,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:13,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:13,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:13,615 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:13,615 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:13,615 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:13,615 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:13,615 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:13,617 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,617 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:13,617 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:13,617 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:13,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,621 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,625 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:13,625 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:13,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:13,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:13,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:13,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:13,625 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,625 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,625 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,625 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:13,625 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:13,625 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:13,625 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:13,625 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:13,626 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,626 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:13,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567713641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:13,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567713643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:13,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52150 deadline: 1682567713652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:13,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567713653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:13,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567713655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,726 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:13,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:13,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:13,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:13,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:13,729 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:13,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:13,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:13,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:13,730 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,730 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,733 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:13,733 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:13,734 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,734 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:13,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:13,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:13,735 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:13,735 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:13,735 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:13,736 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:13,736 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:13,736 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:13,736 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:13,736 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:13,736 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:13,736 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:13,736 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:13,736 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,736 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:13,736 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:13,737 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/c5fd75961933470990d4b94c6ebf7c13 2023-04-27 03:54:13,737 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:13,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:13,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:13,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:13,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:13,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71a46920[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:13,740 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:13,740 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,742 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,743 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:13,743 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:13,743 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:13,743 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,743 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:13,743 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/799b9c0b636e4d6da0d33ec2e2dda0cb as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/799b9c0b636e4d6da0d33ec2e2dda0cb 2023-04-27 03:54:13,744 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,744 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:13,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:13,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:13,744 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:13,744 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:13,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:13,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:13,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:13,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:13,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:13,747 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:13,747 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:13,747 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:13,747 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:13,748 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:13,748 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/799b9c0b636e4d6da0d33ec2e2dda0cb, entries=150, sequenceid=15, filesize=25.1 K 2023-04-27 03:54:13,749 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/f437fc8d4fd04f9a931f64fbbf76e4aa as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/f437fc8d4fd04f9a931f64fbbf76e4aa 2023-04-27 03:54:13,750 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:13,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:13,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:13,751 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,752 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,752 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:13,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:13,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:13,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:13,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,753 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:13,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:13,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,757 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:13,757 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:13,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:13,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:13,758 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:13,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:13,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:13,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:13,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,758 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:13,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:13,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:13,758 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:13,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:13,759 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/f437fc8d4fd04f9a931f64fbbf76e4aa, entries=150, sequenceid=15, filesize=11.6 K 2023-04-27 03:54:13,760 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/c5fd75961933470990d4b94c6ebf7c13 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/c5fd75961933470990d4b94c6ebf7c13 2023-04-27 03:54:13,766 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/c5fd75961933470990d4b94c6ebf7c13, entries=150, sequenceid=15, filesize=11.6 K 2023-04-27 03:54:13,767 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=140.89 KB/144270 for d66a88af1ba39e44423282d0b046d5b3 in 1352ms, sequenceid=15, compaction requested=false 2023-04-27 03:54:13,767 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-27 03:54:13,768 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:13,858 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:13,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:13,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:13,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:13,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:13,863 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:13,863 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:13,864 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:13,864 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:13,864 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,864 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,866 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:13,866 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:13,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:13,867 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:13,867 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:13,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:13,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:13,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:13,868 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:13,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:13,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:13,868 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:13,868 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:13,868 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:13,868 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:13,868 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:13,868 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:13,868 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:13,869 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d66a88af1ba39e44423282d0b046d5b3 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-27 03:54:13,869 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=A 2023-04-27 03:54:13,869 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:13,869 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=B 2023-04-27 03:54:13,869 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:13,869 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=C 2023-04-27 03:54:13,869 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:13,899 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042788dfa8e4b9204dca8d70094235bc4b0c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042788dfa8e4b9204dca8d70094235bc4b0c 2023-04-27 03:54:13,929 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=52, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/964d0f30c5a4487980ed4a60ee95adcb 2023-04-27 03:54:13,950 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/060b290ce2444baf903cd48fc3f55a69 2023-04-27 03:54:13,978 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/a745bc211a4442e4a8136783cf4b4ed6 2023-04-27 03:54:13,986 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/964d0f30c5a4487980ed4a60ee95adcb as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/964d0f30c5a4487980ed4a60ee95adcb 2023-04-27 03:54:13,991 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/964d0f30c5a4487980ed4a60ee95adcb, entries=150, sequenceid=52, filesize=25.1 K 2023-04-27 03:54:13,992 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/060b290ce2444baf903cd48fc3f55a69 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/060b290ce2444baf903cd48fc3f55a69 2023-04-27 03:54:14,002 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/060b290ce2444baf903cd48fc3f55a69, entries=150, sequenceid=52, filesize=11.6 K 2023-04-27 03:54:14,003 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/a745bc211a4442e4a8136783cf4b4ed6 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/a745bc211a4442e4a8136783cf4b4ed6 2023-04-27 03:54:14,007 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/a745bc211a4442e4a8136783cf4b4ed6, entries=150, sequenceid=52, filesize=11.6 K 2023-04-27 03:54:14,008 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=0 B/0 for d66a88af1ba39e44423282d0b046d5b3 in 140ms, sequenceid=52, compaction requested=false 2023-04-27 03:54:14,015 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:14,016 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:14,016 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:14,016 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:14,016 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,016 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:14,016 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:14,018 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,018 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:14,018 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,018 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:14,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:14,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:14,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:14,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58bcc4ac[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:14,020 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:14,020 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,021 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,022 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:14,022 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:14,022 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:14,022 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,022 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:14,023 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,023 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:14,023 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:14,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:14,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:14,023 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:14,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:14,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:14,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:14,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:14,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:14,026 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:14,026 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:14,026 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:14,026 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:14,026 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:14,029 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,029 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:14,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:14,029 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:14,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,030 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:14,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:14,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:14,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:14,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,031 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:14,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,034 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:14,034 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:14,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:14,034 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:14,034 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,034 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:14,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:14,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:14,034 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,035 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:14,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:14,035 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:14,035 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:14,035 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,035 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,035 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:14,135 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:14,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:14,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:14,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:14,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:14,140 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:14,140 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:14,140 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:14,140 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:14,140 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,141 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,142 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:14,142 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:14,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:14,142 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,142 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:14,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:14,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:14,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:14,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:14,144 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:14,144 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:14,144 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:14,145 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:14,145 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:14,145 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:14,145 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:14,145 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:14,145 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:14,145 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:14,145 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:14,145 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,145 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:14,146 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:14,147 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,147 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:14,147 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,147 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:14,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:14,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@500522f6[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:14,150 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:14,151 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,153 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,153 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:14,153 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:14,153 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:14,153 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,153 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:14,155 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:14,155 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:14,155 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:14,155 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:14,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:14,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:14,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:14,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:14,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:14,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:14,158 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:14,159 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:14,159 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:14,159 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:14,159 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:14,161 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:14,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:14,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:14,162 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:14,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,163 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:14,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,167 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,167 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,167 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,167 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,167 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:14,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:14,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:14,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:14,167 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,167 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,167 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,167 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:14,167 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:14,168 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,168 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:14,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,168 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:14,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:14,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:14,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:14,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:14,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:14,267 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:14,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:14,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:14,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:14,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:14,272 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:14,272 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:14,272 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:14,272 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:14,273 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,273 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,275 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:14,275 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:14,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:14,275 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,275 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:14,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:14,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:14,277 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:14,277 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:14,277 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:14,277 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:14,277 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:14,277 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:14,277 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:14,277 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:14,277 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:14,277 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:14,277 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:14,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:14,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:14,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:14,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:14,282 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,282 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:14,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:14,282 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:14,282 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:14,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:14,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:14,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:14,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64ef43b6[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:14,284 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:14,284 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,285 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,286 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:14,286 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:14,286 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:14,288 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,288 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:14,290 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,290 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:14,290 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:14,290 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:14,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:14,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:14,293 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:14,293 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:14,293 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:14,293 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:14,293 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:14,295 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,295 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:14,295 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:14,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:14,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,295 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:14,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:14,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:14,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:14,297 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:14,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,300 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:14,300 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:14,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:14,300 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,300 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,300 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:14,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:14,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:14,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:14,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:14,300 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,301 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:14,301 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:14,301 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,301 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,301 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,301 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:14,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:14,401 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:14,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:14,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:14,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:14,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:14,405 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:14,405 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:14,405 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:14,405 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:14,406 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,406 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,407 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:14,407 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:14,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:14,408 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,408 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:14,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:14,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:14,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:14,409 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:14,409 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:14,409 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:14,409 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:14,410 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:14,410 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:14,410 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:14,410 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:14,410 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:14,410 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:14,410 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:14,410 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,410 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:14,410 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:14,412 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,412 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:14,413 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,413 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:14,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:14,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3650cb62[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:14,415 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:14,415 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,416 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,416 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:14,416 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:14,416 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:14,416 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,416 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:14,417 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:14,417 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:14,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:14,417 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:14,418 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:14,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:14,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:14,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:14,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:14,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:14,421 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:14,421 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:14,421 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:14,421 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:14,421 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:14,423 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:14,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:14,423 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,423 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:14,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:14,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:14,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:14,424 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:14,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,425 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,428 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:14,428 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:14,428 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,428 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,428 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,428 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,429 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:14,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:14,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:14,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:14,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:14,429 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,429 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,429 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:14,429 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:14,429 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:14,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:14,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:14,529 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:14,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:14,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:14,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:14,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:14,534 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:14,534 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:14,534 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:14,534 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:14,534 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,535 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:14,536 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:14,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:14,536 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,536 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:14,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:14,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,537 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:14,544 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:14,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:14,544 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:14,544 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:14,544 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:14,556 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:14,556 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:14,568 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:14,568 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:14,568 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:14,568 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:14,568 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:14,568 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:14,568 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,568 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:14,568 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:14,570 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,570 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:14,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:14,571 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,571 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:14,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:14,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:14,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:14,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:14,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@877f0a0[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:14,573 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:14,573 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,574 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,574 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:14,574 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:14,574 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:14,574 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,574 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:14,576 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:14,576 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:14,577 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:14,577 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:14,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:14,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:14,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:14,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:14,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:14,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:14,580 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:14,580 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:14,580 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:14,580 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:14,580 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:14,581 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,582 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,582 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:14,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:14,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:14,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:14,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:14,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:14,583 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:14,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,584 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:14,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,587 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:14,587 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:14,587 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:14,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:14,587 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:14,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:14,587 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:14,587 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:14,587 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:14,587 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,587 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,587 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,587 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,587 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:14,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:14,687 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:14,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:14,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:14,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:14,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:14,692 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:14,692 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:14,692 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:14,692 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:14,693 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,693 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,694 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:14,694 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:14,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:14,694 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,695 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:14,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,695 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d66a88af1ba39e44423282d0b046d5b3 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:54:14,695 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=A 2023-04-27 03:54:14,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:14,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:14,695 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:14,696 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=B 2023-04-27 03:54:14,696 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:14,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:14,696 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=C 2023-04-27 03:54:14,696 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:14,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:14,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:14,700 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:14,700 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:14,700 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:14,704 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:14,704 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:14,705 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:14,705 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:14,705 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:14,705 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:14,705 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:14,705 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:14,705 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,705 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:14,705 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:14,708 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,708 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,709 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:14,710 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:14,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:14,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:14,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:14,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:14,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:14,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@711c61ca[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:14,713 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:14,713 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,715 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,715 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:14,715 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:14,715 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:14,715 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,715 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:14,721 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:14,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:14,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:14,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:14,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:14,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:14,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:14,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:14,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:14,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:14,727 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:14,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:14,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:14,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:14,727 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:14,728 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042776f1ac940c374cd783fa4ff54ef04b7d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042776f1ac940c374cd783fa4ff54ef04b7d 2023-04-27 03:54:14,730 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,731 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:14,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,731 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:14,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:14,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:14,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:14,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:14,732 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:14,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,734 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:14,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,737 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:14,737 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:14,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:14,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:14,737 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,737 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,737 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,737 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,737 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,737 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,737 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:14,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:14,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:14,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:14,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:14,738 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:14,738 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:14,739 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=66, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/b628d996fd6647b7bcabe35da1a1d79b 2023-04-27 03:54:14,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1682567654697, max timestamp=1682567654738Num uniques 859; 2023-04-27 03:54:14,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-27 03:54:14,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1682567654697, max timestamp=1682567654738Num uniques 859; 2023-04-27 03:54:14,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-27 03:54:14,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1682567654697, max timestamp=1682567654738Num uniques 859; 2023-04-27 03:54:14,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-27 03:54:14,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:14,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567714743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:14,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567714744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:14,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567714747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:14,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567714748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:14,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52150 deadline: 1682567714749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,761 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/97d8cc4120f44b16981de814a6ca81f5 2023-04-27 03:54:14,838 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:14,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:14,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:14,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:14,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:14,842 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:14,842 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:14,842 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:14,842 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:14,842 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,842 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,843 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,843 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:14,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:14,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:14,844 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,844 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:14,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:14,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:14,845 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:14,845 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:14,845 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:14,845 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:14,845 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:14,845 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:14,845 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:14,845 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:14,845 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:14,845 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:14,845 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:14,846 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:14,846 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:14,846 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,846 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:14,846 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:14,847 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:14,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:14,847 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:14,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:14,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:14,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:14,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:14,849 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:14,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ff07eac[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:14,849 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,852 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,852 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:14,852 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:14,852 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:14,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:14,853 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,853 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:14,853 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:14,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:14,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:14,853 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:14,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:14,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567714851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:14,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567714852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:14,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567714854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:14,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:14,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:14,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567714860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:14,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52150 deadline: 1682567714860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:14,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:14,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:14,863 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:14,863 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:14,863 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:14,863 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:14,863 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:14,865 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,865 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:14,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:14,865 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:14,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,866 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:14,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:14,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:14,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:14,867 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,869 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:14,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:14,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:14,869 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,870 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:14,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:14,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:14,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:14,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:14,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:14,870 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:14,870 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:14,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:14,970 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:14,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:14,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:14,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:14,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:14,974 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:14,974 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:14,974 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:14,974 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:14,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,977 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:14,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:14,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:14,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,978 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:14,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:14,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:14,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:14,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:14,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:14,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:14,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:14,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:14,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:14,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:14,983 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:14,983 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:14,983 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:14,984 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:14,984 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:14,984 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:14,984 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,984 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:14,984 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:14,985 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:14,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:14,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:14,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:14,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,987 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:14,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:14,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:14,988 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:14,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f4bb3eb[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:14,988 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,989 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:14,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:14,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:14,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:14,989 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,989 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:14,991 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,991 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:14,991 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:14,991 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:14,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:14,994 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:14,994 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:14,994 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:14,994 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:14,994 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:14,998 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:14,998 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,998 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:14,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:14,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:14,999 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:15,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:15,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:15,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,001 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:15,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:15,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,003 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,003 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:15,003 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:15,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:15,003 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:15,004 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,004 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:15,004 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:15,004 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:15,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:15,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:15,003 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:15,003 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:15,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:15,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:15,004 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:15,004 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:15,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:15,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:15,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:15,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567715057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:15,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567715057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:15,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567715059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:15,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52150 deadline: 1682567715065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:15,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567715065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,104 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:15,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:15,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:15,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:15,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:15,116 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:15,116 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:15,116 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:15,116 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:15,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:15,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:15,118 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:15,118 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:15,118 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,119 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:15,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:15,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:15,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:15,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:15,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:15,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:15,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:15,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:15,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:15,122 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:15,122 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:15,122 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:15,122 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:15,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:15,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:15,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:15,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:15,123 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,123 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:15,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:15,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:15,124 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:15,124 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:15,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:15,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:15,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:15,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:15,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:15,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@688eba52[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:15,126 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:15,126 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:15,129 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:15,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:15,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:15,129 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:15,129 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:15,129 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:15,129 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,129 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:15,130 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:15,130 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:15,130 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,130 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:15,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:15,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:15,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:15,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:15,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:15,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:15,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:15,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:15,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:15,136 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:15,136 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:15,136 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:15,136 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:15,136 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:15,141 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:15,141 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:15,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:15,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:15,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:15,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:15,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:15,142 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:15,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:15,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:15,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:15,143 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:15,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:15,144 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:15,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:15,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,154 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:15,154 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:15,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:15,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:15,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:15,154 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:15,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:15,154 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,154 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:15,154 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:15,154 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,154 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:15,154 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:15,154 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:15,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:15,154 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:15,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:15,154 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:15,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:15,184 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/b7ce1d66c36c4fb7988a3031334cc076 2023-04-27 03:54:15,189 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/b628d996fd6647b7bcabe35da1a1d79b as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/b628d996fd6647b7bcabe35da1a1d79b 2023-04-27 03:54:15,193 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/b628d996fd6647b7bcabe35da1a1d79b, entries=350, sequenceid=66, filesize=52.3 K 2023-04-27 03:54:15,196 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/97d8cc4120f44b16981de814a6ca81f5 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/97d8cc4120f44b16981de814a6ca81f5 2023-04-27 03:54:15,201 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/97d8cc4120f44b16981de814a6ca81f5, entries=350, sequenceid=66, filesize=20.8 K 2023-04-27 03:54:15,204 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/b7ce1d66c36c4fb7988a3031334cc076 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/b7ce1d66c36c4fb7988a3031334cc076 2023-04-27 03:54:15,211 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/b7ce1d66c36c4fb7988a3031334cc076, entries=150, sequenceid=66, filesize=11.6 K 2023-04-27 03:54:15,212 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=154.31 KB/158010 for d66a88af1ba39e44423282d0b046d5b3 in 517ms, sequenceid=66, compaction requested=true 2023-04-27 03:54:15,212 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:15,212 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:15,212 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:15,212 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:15,212 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:15,212 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:15,213 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 45022 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:15,213 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): d66a88af1ba39e44423282d0b046d5b3/B is initiating minor compaction (all files) 2023-04-27 03:54:15,213 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 105002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:15,213 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d66a88af1ba39e44423282d0b046d5b3/B in TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:15,213 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): d66a88af1ba39e44423282d0b046d5b3/A is initiating minor compaction (all files) 2023-04-27 03:54:15,214 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/f437fc8d4fd04f9a931f64fbbf76e4aa, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/060b290ce2444baf903cd48fc3f55a69, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/97d8cc4120f44b16981de814a6ca81f5] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp, totalSize=44.0 K 2023-04-27 03:54:15,214 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d66a88af1ba39e44423282d0b046d5b3/A in TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:15,214 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/799b9c0b636e4d6da0d33ec2e2dda0cb, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/964d0f30c5a4487980ed4a60ee95adcb, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/b628d996fd6647b7bcabe35da1a1d79b] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp, totalSize=102.5 K 2023-04-27 03:54:15,215 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting f437fc8d4fd04f9a931f64fbbf76e4aa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682567652404 2023-04-27 03:54:15,215 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 799b9c0b636e4d6da0d33ec2e2dda0cb, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682567652404 2023-04-27 03:54:15,215 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 060b290ce2444baf903cd48fc3f55a69, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1682567652477 2023-04-27 03:54:15,215 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 964d0f30c5a4487980ed4a60ee95adcb, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1682567652477 2023-04-27 03:54:15,216 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 97d8cc4120f44b16981de814a6ca81f5, keycount=350, bloomtype=ROW, size=20.8 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1682567654681 2023-04-27 03:54:15,216 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting b628d996fd6647b7bcabe35da1a1d79b, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1682567654681 2023-04-27 03:54:15,225 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): d66a88af1ba39e44423282d0b046d5b3#B#compaction#122 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:15,251 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): d66a88af1ba39e44423282d0b046d5b3#A#compaction#123 average throughput is 0.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:15,255 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:15,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:15,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:15,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:15,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:15,264 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:15,264 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:15,264 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:15,264 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:15,265 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:15,265 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:15,266 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:15,266 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:15,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:15,266 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,267 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:15,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:15,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:15,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:15,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:15,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:15,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:15,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:15,276 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:15,276 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:15,276 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:15,277 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:15,277 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:15,277 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:15,277 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:15,277 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d66a88af1ba39e44423282d0b046d5b3 3/3 column families, dataSize=154.31 KB heapSize=379.13 KB 2023-04-27 03:54:15,278 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=A 2023-04-27 03:54:15,278 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:15,278 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=B 2023-04-27 03:54:15,278 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:15,278 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=C 2023-04-27 03:54:15,278 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:15,282 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/42b8796756d44e55a37c1f9e67d28071 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/42b8796756d44e55a37c1f9e67d28071 2023-04-27 03:54:15,287 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d66a88af1ba39e44423282d0b046d5b3/B of d66a88af1ba39e44423282d0b046d5b3 into 42b8796756d44e55a37c1f9e67d28071(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:15,287 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:15,287 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., storeName=d66a88af1ba39e44423282d0b046d5b3/B, priority=13, startTime=1682567655212; duration=0sec 2023-04-27 03:54:15,287 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:15,287 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:15,289 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:15,289 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): d66a88af1ba39e44423282d0b046d5b3/C is initiating minor compaction (all files) 2023-04-27 03:54:15,289 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d66a88af1ba39e44423282d0b046d5b3/C in TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:15,290 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/c5fd75961933470990d4b94c6ebf7c13, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/a745bc211a4442e4a8136783cf4b4ed6, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/b7ce1d66c36c4fb7988a3031334cc076] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp, totalSize=34.8 K 2023-04-27 03:54:15,290 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting c5fd75961933470990d4b94c6ebf7c13, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682567652404 2023-04-27 03:54:15,290 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting a745bc211a4442e4a8136783cf4b4ed6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1682567652477 2023-04-27 03:54:15,291 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting b7ce1d66c36c4fb7988a3031334cc076, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1682567654685 2023-04-27 03:54:15,301 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): d66a88af1ba39e44423282d0b046d5b3#C#compaction#125 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:15,325 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/81630c9d75e14440889a470c0b95bddd as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/81630c9d75e14440889a470c0b95bddd 2023-04-27 03:54:15,331 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d66a88af1ba39e44423282d0b046d5b3/C of d66a88af1ba39e44423282d0b046d5b3 into 81630c9d75e14440889a470c0b95bddd(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:15,332 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:15,332 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., storeName=d66a88af1ba39e44423282d0b046d5b3/C, priority=13, startTime=1682567655212; duration=0sec 2023-04-27 03:54:15,332 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:15,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:15,363 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:15,372 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:15,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567715368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,372 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:15,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52150 deadline: 1682567715369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:15,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567715370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:15,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567715370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:15,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567715372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:15,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567715474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:15,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567715474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:15,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567715480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:15,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567715680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:15,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567715681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:15,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567715683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,721 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427537675995a4f4c9caef3a8880a628e5f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427537675995a4f4c9caef3a8880a628e5f 2023-04-27 03:54:15,734 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/e6c0f4136fff4b7d99b76687617ef264 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/e6c0f4136fff4b7d99b76687617ef264 2023-04-27 03:54:15,739 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d66a88af1ba39e44423282d0b046d5b3/A of d66a88af1ba39e44423282d0b046d5b3 into e6c0f4136fff4b7d99b76687617ef264(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:15,739 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:15,739 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., storeName=d66a88af1ba39e44423282d0b046d5b3/A, priority=13, startTime=1682567655212; duration=0sec 2023-04-27 03:54:15,739 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:15,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:15,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52150 deadline: 1682567715877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:15,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567715878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:15,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567715985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:15,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567715985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:15,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:15,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567715986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,128 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=92, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/90beec4eb0bd4e52a1a794e3720913ff 2023-04-27 03:54:16,141 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/991219af4c9b46388020cece1ca5fbe9 2023-04-27 03:54:16,157 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/06dc6270b31e498fad233379f8fb6091 2023-04-27 03:54:16,162 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/90beec4eb0bd4e52a1a794e3720913ff as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/90beec4eb0bd4e52a1a794e3720913ff 2023-04-27 03:54:16,168 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/90beec4eb0bd4e52a1a794e3720913ff, entries=150, sequenceid=92, filesize=25.1 K 2023-04-27 03:54:16,169 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/991219af4c9b46388020cece1ca5fbe9 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/991219af4c9b46388020cece1ca5fbe9 2023-04-27 03:54:16,173 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/991219af4c9b46388020cece1ca5fbe9, entries=150, sequenceid=92, filesize=11.6 K 2023-04-27 03:54:16,175 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/06dc6270b31e498fad233379f8fb6091 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/06dc6270b31e498fad233379f8fb6091 2023-04-27 03:54:16,179 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/06dc6270b31e498fad233379f8fb6091, entries=150, sequenceid=92, filesize=11.6 K 2023-04-27 03:54:16,180 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~379.08 KB/388176, currentSize=46.96 KB/48090 for d66a88af1ba39e44423282d0b046d5b3 in 903ms, sequenceid=92, compaction requested=false 2023-04-27 03:54:16,180 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:16,180 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:16,180 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:16,180 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:16,180 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,184 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:16,184 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:16,186 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,186 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:16,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:16,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:16,187 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:16,187 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:16,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:16,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:16,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:16,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:16,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:16,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a8e94b8[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:16,188 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:16,188 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:16,189 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:16,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:16,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:16,190 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:16,190 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:16,190 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:16,192 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:16,192 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:16,192 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,192 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:16,193 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:16,193 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:16,193 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:16,193 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:16,193 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:16,193 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:16,193 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:16,195 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,195 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:16,195 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:16,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:16,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:16,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:16,196 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:16,197 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:16,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:16,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:16,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:16,200 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:16,200 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:16,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:16,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:16,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:16,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:16,200 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,200 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:16,201 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:16,201 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:16,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:16,201 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:16,201 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:16,201 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,201 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:16,201 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:16,201 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:16,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:16,301 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:16,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:16,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:16,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:16,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:16,305 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:16,305 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:16,305 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:16,305 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:16,305 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,306 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:16,307 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:16,307 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:16,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:16,307 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,307 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:16,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:16,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:16,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:16,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:16,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:16,308 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:16,309 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:16,309 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:16,309 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:16,309 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:16,309 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:16,309 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:16,309 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d66a88af1ba39e44423282d0b046d5b3 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:54:16,309 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=A 2023-04-27 03:54:16,309 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:16,310 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=B 2023-04-27 03:54:16,310 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:16,310 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=C 2023-04-27 03:54:16,310 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:16,328 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427f3fb1a725abd4f91be12bba2b10cb9a0 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427f3fb1a725abd4f91be12bba2b10cb9a0 2023-04-27 03:54:16,340 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=105, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/1786be032664411dbd01f7729d7f63d2 2023-04-27 03:54:16,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:16,489 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:16,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682567656496, max timestamp=1682567656507Num uniques 859; 2023-04-27 03:54:16,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682567656496, max timestamp=1682567656507Num uniques 859; 2023-04-27 03:54:16,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-27 03:54:16,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-27 03:54:16,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682567656496, max timestamp=1682567656507Num uniques 859; 2023-04-27 03:54:16,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-27 03:54:16,526 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:16,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567716525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:16,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567716526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:16,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567716526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:16,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567716628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:16,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567716628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:16,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567716630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,754 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/428beccef25046b6b484be6b0d325449 2023-04-27 03:54:16,768 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/5e16a187dc8b4957ae3aab370f24be74 2023-04-27 03:54:16,773 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/1786be032664411dbd01f7729d7f63d2 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/1786be032664411dbd01f7729d7f63d2 2023-04-27 03:54:16,777 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/1786be032664411dbd01f7729d7f63d2, entries=150, sequenceid=105, filesize=25.1 K 2023-04-27 03:54:16,778 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/428beccef25046b6b484be6b0d325449 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/428beccef25046b6b484be6b0d325449 2023-04-27 03:54:16,788 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/428beccef25046b6b484be6b0d325449, entries=150, sequenceid=105, filesize=11.6 K 2023-04-27 03:54:16,789 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/5e16a187dc8b4957ae3aab370f24be74 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/5e16a187dc8b4957ae3aab370f24be74 2023-04-27 03:54:16,793 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/5e16a187dc8b4957ae3aab370f24be74, entries=150, sequenceid=105, filesize=11.6 K 2023-04-27 03:54:16,794 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for d66a88af1ba39e44423282d0b046d5b3 in 485ms, sequenceid=105, compaction requested=true 2023-04-27 03:54:16,795 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:16,795 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:16,795 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:16,795 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:16,795 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,795 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:16,795 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:16,797 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,797 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:16,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:16,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:16,798 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:16,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:16,798 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:16,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:16,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:16,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:16,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:16,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16e7805b[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:16,799 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:16,799 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:16,800 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:16,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:16,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:16,801 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:16,801 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:16,801 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:16,801 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,801 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:16,802 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:16,802 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,802 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:16,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:16,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:16,802 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:16,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:16,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:16,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:16,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:16,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:16,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:16,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:16,805 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:16,805 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:16,805 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:16,805 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:16,805 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:16,806 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,806 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:16,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:16,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,806 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:16,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:16,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:16,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:16,807 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:16,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:16,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:16,808 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:16,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:16,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,811 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:16,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:16,811 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:16,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:16,811 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,811 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:16,811 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:16,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:16,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:16,811 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:16,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:16,812 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:16,812 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:16,811 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:16,811 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:16,812 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:16,812 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:16,836 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d66a88af1ba39e44423282d0b046d5b3 3/3 column families, dataSize=147.60 KB heapSize=410.77 KB 2023-04-27 03:54:16,836 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=A 2023-04-27 03:54:16,837 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:16,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=B 2023-04-27 03:54:16,837 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:16,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=C 2023-04-27 03:54:16,837 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:16,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:16,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567716840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:16,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567716841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:16,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567716846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,850 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427d81bee79475249888090a1d00c7ae067 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427d81bee79475249888090a1d00c7ae067 2023-04-27 03:54:16,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:16,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52150 deadline: 1682567716879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:16,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567716882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,912 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:16,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:16,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:16,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:16,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:16,915 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:16,915 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:16,915 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:16,915 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:16,916 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,916 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:16,917 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,917 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:16,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:16,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:16,917 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,917 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:16,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:16,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:16,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:16,918 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:16,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:16,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:16,919 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:16,919 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:16,919 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:16,919 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:16,919 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:16,919 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:16,919 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:16,919 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:16,919 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:16,919 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:16,919 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:16,919 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,919 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:16,919 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:16,921 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,921 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:16,921 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:16,921 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:16,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:16,923 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:16,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1431bd87[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:16,923 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:16,924 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:16,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:16,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:16,924 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:16,924 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:16,924 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,924 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:16,924 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:16,927 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,927 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:16,927 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:16,927 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:16,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:16,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:16,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:16,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:16,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:16,929 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:16,929 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:16,929 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:16,929 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:16,929 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:16,931 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,931 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:16,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:16,931 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:16,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:16,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:16,932 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:16,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:16,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:16,933 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:16,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:16,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:16,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,936 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:16,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:16,936 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,936 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:16,936 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:16,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:16,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:16,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:16,936 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:16,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:16,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:16,937 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:16,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:16,937 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,937 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:16,937 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:16,937 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:16,937 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:16,937 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:16,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:16,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567716947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:16,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567716948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:16,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:16,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567716951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,037 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:17,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:17,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:17,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:17,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:17,041 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:17,041 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:17,041 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:17,041 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:17,041 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,041 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,043 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,043 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:17,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:17,044 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,044 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:17,045 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:17,045 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:17,045 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:17,045 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:17,045 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:17,045 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:17,045 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:17,045 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:17,045 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:17,045 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:17,045 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:17,045 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:17,045 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:17,045 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,045 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:17,045 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:17,047 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,047 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:17,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:17,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:17,047 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,047 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:17,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:17,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:17,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:17,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@970c979[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:17,049 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:17,049 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,050 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,051 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:17,051 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:17,051 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:17,051 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,051 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:17,052 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:17,052 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,052 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:17,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:17,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:17,052 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:17,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:17,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:17,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:17,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:17,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:17,055 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:17,055 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:17,055 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:17,055 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:17,055 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:17,058 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,058 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,058 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:17,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:17,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:17,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:17,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:17,059 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:17,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:17,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,060 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:17,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,062 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,062 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,062 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,062 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:17,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:17,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:17,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:17,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:17,062 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:17,062 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:17,063 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:17,063 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:17,062 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:17,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:17,063 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:17,063 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,063 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:17,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567717150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:17,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567717150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:17,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567717154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,163 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:17,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:17,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:17,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:17,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:17,167 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:17,167 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:17,167 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:17,167 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:17,168 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,168 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,169 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:17,169 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:17,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:17,170 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,170 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:17,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:17,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,171 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:17,171 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:17,171 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:17,171 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:17,171 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:17,171 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:17,171 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:17,171 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:17,171 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:17,171 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:17,171 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:17,172 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:17,172 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:17,172 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:17,172 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,172 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:17,172 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:17,173 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,173 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:17,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:17,173 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,174 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:17,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:17,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:17,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:17,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:17,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755b7bb[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:17,175 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:17,175 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,177 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,177 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,177 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:17,178 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:17,178 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:17,178 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:17,180 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:17,180 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:17,180 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,180 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:17,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:17,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:17,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:17,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:17,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:17,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:17,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:17,182 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:17,182 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:17,183 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:17,183 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:17,183 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:17,184 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,184 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:17,184 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:17,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:17,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:17,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:17,185 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:17,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:17,186 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:17,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,188 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,188 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,188 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:17,188 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:17,188 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:17,188 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,188 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,188 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,188 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:17,188 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:17,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:17,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:17,189 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:17,189 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:17,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:17,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:17,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:17,255 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=142, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/eacbc85e11f54c92ad93d692bf9246c8 2023-04-27 03:54:17,289 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:17,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:17,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:17,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:17,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:17,292 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:17,292 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:17,292 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:17,292 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:17,293 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,293 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,295 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:17,295 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:17,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:17,295 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,295 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:17,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:17,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:17,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:17,297 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:17,297 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:17,297 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:17,297 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:17,297 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:17,297 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:17,297 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:17,297 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:17,297 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:17,297 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:17,297 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:17,297 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:17,297 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,297 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:17,297 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:17,299 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,299 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:17,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:17,299 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,299 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:17,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:17,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:17,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:17,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:17,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d9a0a52[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:17,301 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:17,301 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,302 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,302 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:17,302 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:17,302 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:17,302 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,302 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:17,303 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,303 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:17,304 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:17,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,304 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:17,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:17,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:17,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:17,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:17,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:17,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:17,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:17,306 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:17,306 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:17,306 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:17,306 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:17,306 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:17,309 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,309 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:17,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:17,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,309 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:17,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:17,309 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:17,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:17,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:17,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,311 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:17,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,320 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:17,320 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:17,320 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:17,320 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:17,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:17,320 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:17,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:17,320 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,320 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,321 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,321 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:17,321 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,321 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:17,321 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:17,421 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:17,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:17,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:17,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:17,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:17,425 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:17,425 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:17,425 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:17,425 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:17,426 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,426 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,427 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:17,427 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:17,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:17,428 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,428 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:17,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:17,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,429 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:17,429 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:17,429 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:17,429 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:17,429 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:17,429 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:17,429 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:17,430 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:17,430 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:17,430 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:17,430 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:17,430 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:17,430 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:17,430 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:17,430 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,430 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:17,430 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:17,433 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,433 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:17,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:17,433 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,433 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:17,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:17,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:17,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:17,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:17,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34de57e3[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:17,435 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:17,435 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,436 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,437 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:17,437 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:17,437 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:17,437 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,437 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:17,438 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:17,438 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:17,438 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:17,438 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:17,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:17,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:17,441 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:17,441 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:17,441 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:17,441 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:17,441 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:17,444 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,444 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,444 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:17,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:17,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:17,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:17,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:17,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:17,445 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:17,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,446 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:17,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,454 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,454 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:17,454 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:17,454 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:17,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:17,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:17,454 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:17,454 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,455 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,455 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,455 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,455 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,455 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:17,455 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:17,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:17,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:17,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:17,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:17,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567717453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:17,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567717453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:17,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567717458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,555 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:17,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:17,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:17,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:17,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:17,558 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:17,558 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:17,558 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:17,558 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:17,559 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,559 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,561 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,561 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:17,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:17,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:17,561 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,561 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:17,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:17,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:17,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:17,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:17,562 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:17,562 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:17,562 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:17,562 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:17,563 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:17,563 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:17,563 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:17,563 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:17,563 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:17,563 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:17,563 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:17,563 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,563 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:17,563 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:17,564 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,564 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:17,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:17,564 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,565 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:17,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:17,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:17,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:17,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:17,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b6a7b00[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:17,566 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:17,566 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,567 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,567 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:17,567 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:17,567 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:17,567 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,567 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:17,569 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,569 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:17,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:17,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:17,569 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:17,569 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:17,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:17,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:17,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:17,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:17,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:17,571 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:17,571 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:17,571 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:17,571 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:17,571 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:17,573 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,573 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:17,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:17,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,573 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:17,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:17,573 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:17,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:17,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:17,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,575 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:17,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,577 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,577 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,577 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:17,577 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:17,577 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:17,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:17,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:17,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:17,577 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:17,577 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,577 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,578 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,578 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:17,578 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:17,578 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:17,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:17,669 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/48656692b4df43469050053de8e079c6 2023-04-27 03:54:17,678 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:17,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:17,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:17,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:17,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:17,682 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:17,683 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:17,683 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:17,683 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:17,683 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,683 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,684 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:17,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:17,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:17,684 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,685 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,685 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:17,686 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:17,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:17,686 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:17,686 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:17,686 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:17,687 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:17,687 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:17,687 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:17,687 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:17,687 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:17,687 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:17,687 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:17,687 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:17,687 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,687 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:17,687 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:17,689 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,689 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:17,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:17,690 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,690 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:17,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:17,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1128fbb9[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:17,692 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:17,692 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,693 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,693 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:17,693 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:17,693 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:17,694 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:17,694 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:17,695 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:17,695 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,695 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:17,695 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:17,695 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:17,695 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:17,695 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:17,695 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:17,695 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:17,696 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,696 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,697 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:17,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:17,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:17,697 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:17,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,698 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:17,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,702 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,702 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:17,702 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:17,702 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:17,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:17,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:17,702 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:17,702 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:17,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:17,702 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:17,702 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:17,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:17,702 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,702 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,702 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,703 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,802 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:17,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:17,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:17,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:17,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:17,806 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:17,806 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:17,806 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:17,806 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:17,807 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,807 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,808 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,808 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:17,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:17,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:17,808 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,808 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:17,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:17,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:17,809 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:17,809 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:17,809 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:17,809 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:17,809 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:17,809 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:17,810 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:17,810 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:17,810 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:17,810 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:17,810 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:17,810 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:17,810 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:17,810 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,810 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:17,810 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:17,814 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,814 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:17,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:17,815 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,815 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:17,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:17,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:17,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:17,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:17,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c1dbb54[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:17,816 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:17,816 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,818 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,819 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:17,819 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:17,819 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:17,819 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,819 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:17,820 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:17,820 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:17,820 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,820 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:17,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:17,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:17,823 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:17,823 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:17,823 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:17,823 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:17,823 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:17,824 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,824 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:17,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:17,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,824 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:17,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:17,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:17,825 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:17,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:17,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,826 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:17,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,828 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,828 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,828 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:17,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:17,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:17,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:17,828 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:17,828 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,828 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,829 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,829 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:17,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:17,828 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:17,829 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:17,829 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:17,829 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:17,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:17,929 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:17,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:17,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:17,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:17,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:17,933 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:17,933 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:17,933 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:17,933 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:17,933 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,933 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,940 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:17,940 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:17,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:17,941 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,941 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:17,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:17,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:17,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:17,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:17,942 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:17,942 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:17,942 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:17,942 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:17,942 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:17,942 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:17,942 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:17,942 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:17,942 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:17,942 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:17,943 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:17,943 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,943 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:17,943 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:17,944 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,944 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:17,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:17,944 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,944 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:17,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:17,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:17,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:17,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:17,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a3a987b[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:17,946 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:17,946 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,947 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,947 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:17,947 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:17,947 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:17,947 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,947 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:17,948 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:17,948 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:17,948 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,948 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:17,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:17,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:17,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:17,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:17,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:17,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:17,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:17,951 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:17,951 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:17,951 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:17,951 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:17,951 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:17,954 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,954 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,954 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:17,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:17,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:17,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:17,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:17,954 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:17,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:17,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,956 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:17,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:17,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,958 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:17,958 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:17,958 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:17,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:17,958 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:17,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:17,958 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:17,959 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:17,959 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:17,959 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:17,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:17,959 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,959 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,959 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:17,959 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:17,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:17,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567717960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:17,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567717961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:17,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:17,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567717962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,059 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:18,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:18,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:18,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:18,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:18,064 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:18,064 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:18,064 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:18,064 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:18,065 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,065 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:18,066 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:18,066 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:18,066 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,066 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:18,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:18,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:18,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:18,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:18,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:18,068 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:18,068 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:18,068 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:18,068 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:18,068 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:18,068 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:18,068 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:18,068 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:18,068 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:18,068 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:18,068 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:18,068 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,068 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:18,068 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:18,069 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,069 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:18,070 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,070 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:18,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:18,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42e05447[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:18,072 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:18,072 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,074 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,074 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:18,074 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:18,074 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:18,074 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,074 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:18,075 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:18,075 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:18,075 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:18,075 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:18,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:18,077 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:18,078 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:18,078 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:18,078 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:18,078 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:18,079 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,079 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:18,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,079 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:18,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:18,080 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:18,081 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/62ed9585f56d49489011bb981df1cfb8 2023-04-27 03:54:18,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:18,081 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,084 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,084 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:18,084 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:18,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:18,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:18,084 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:18,084 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:18,084 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:18,084 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:18,084 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:18,084 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:18,084 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,084 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,085 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:18,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:18,086 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/eacbc85e11f54c92ad93d692bf9246c8 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/eacbc85e11f54c92ad93d692bf9246c8 2023-04-27 03:54:18,090 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/eacbc85e11f54c92ad93d692bf9246c8, entries=250, sequenceid=142, filesize=38.9 K 2023-04-27 03:54:18,092 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/48656692b4df43469050053de8e079c6 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/48656692b4df43469050053de8e079c6 2023-04-27 03:54:18,097 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/48656692b4df43469050053de8e079c6, entries=150, sequenceid=142, filesize=11.8 K 2023-04-27 03:54:18,098 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/62ed9585f56d49489011bb981df1cfb8 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/62ed9585f56d49489011bb981df1cfb8 2023-04-27 03:54:18,101 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/62ed9585f56d49489011bb981df1cfb8, entries=150, sequenceid=142, filesize=11.8 K 2023-04-27 03:54:18,102 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~430.64 KB/440976, currentSize=33.54 KB/34350 for d66a88af1ba39e44423282d0b046d5b3 in 1267ms, sequenceid=142, compaction requested=true 2023-04-27 03:54:18,102 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:18,102 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:18,102 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 03:54:18,102 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 03:54:18,102 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:18,102 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:18,104 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 117221 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 03:54:18,104 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47797 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 03:54:18,104 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): d66a88af1ba39e44423282d0b046d5b3/A is initiating minor compaction (all files) 2023-04-27 03:54:18,104 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): d66a88af1ba39e44423282d0b046d5b3/B is initiating minor compaction (all files) 2023-04-27 03:54:18,104 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d66a88af1ba39e44423282d0b046d5b3/A in TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:18,104 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d66a88af1ba39e44423282d0b046d5b3/B in TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:18,104 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/e6c0f4136fff4b7d99b76687617ef264, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/90beec4eb0bd4e52a1a794e3720913ff, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/1786be032664411dbd01f7729d7f63d2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/eacbc85e11f54c92ad93d692bf9246c8] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp, totalSize=114.5 K 2023-04-27 03:54:18,104 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/42b8796756d44e55a37c1f9e67d28071, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/991219af4c9b46388020cece1ca5fbe9, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/428beccef25046b6b484be6b0d325449, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/48656692b4df43469050053de8e079c6] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp, totalSize=46.7 K 2023-04-27 03:54:18,104 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting e6c0f4136fff4b7d99b76687617ef264, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1682567654685 2023-04-27 03:54:18,104 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 42b8796756d44e55a37c1f9e67d28071, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1682567654685 2023-04-27 03:54:18,105 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 90beec4eb0bd4e52a1a794e3720913ff, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1682567654741 2023-04-27 03:54:18,105 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 991219af4c9b46388020cece1ca5fbe9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1682567654741 2023-04-27 03:54:18,105 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 1786be032664411dbd01f7729d7f63d2, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1682567655364 2023-04-27 03:54:18,105 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 428beccef25046b6b484be6b0d325449, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1682567655364 2023-04-27 03:54:18,106 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting eacbc85e11f54c92ad93d692bf9246c8, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682567656524 2023-04-27 03:54:18,106 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 48656692b4df43469050053de8e079c6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682567656524 2023-04-27 03:54:18,121 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): d66a88af1ba39e44423282d0b046d5b3#B#compaction#135 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:18,130 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): d66a88af1ba39e44423282d0b046d5b3#A#compaction#134 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:18,184 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:18,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:18,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:18,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:18,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:18,188 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:18,188 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:18,188 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:18,188 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:18,189 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,189 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:18,191 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:18,191 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:18,192 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:18,192 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:18,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:18,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:18,193 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:18,193 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:18,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:18,193 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:18,193 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:18,193 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:18,194 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:18,194 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:18,194 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:18,194 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:18,194 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d66a88af1ba39e44423282d0b046d5b3 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-27 03:54:18,194 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=A 2023-04-27 03:54:18,194 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:18,194 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=B 2023-04-27 03:54:18,194 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:18,194 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=C 2023-04-27 03:54:18,194 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:18,220 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304272ae4f172315b487d9633135b4def1c44 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272ae4f172315b487d9633135b4def1c44 2023-04-27 03:54:18,230 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=150, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/f7ec5aa9b3bd41b796fd6f744f8d4d96 2023-04-27 03:54:18,247 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/cf41efa3cad64b1980e035e7a02056aa 2023-04-27 03:54:18,558 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/88753208d5d74c6eba0522e2948b3a99 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/88753208d5d74c6eba0522e2948b3a99 2023-04-27 03:54:18,563 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d66a88af1ba39e44423282d0b046d5b3/B of d66a88af1ba39e44423282d0b046d5b3 into 88753208d5d74c6eba0522e2948b3a99(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:18,563 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:18,563 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., storeName=d66a88af1ba39e44423282d0b046d5b3/B, priority=12, startTime=1682567658102; duration=0sec 2023-04-27 03:54:18,563 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:18,563 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 03:54:18,565 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47797 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 03:54:18,565 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): d66a88af1ba39e44423282d0b046d5b3/C is initiating minor compaction (all files) 2023-04-27 03:54:18,565 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d66a88af1ba39e44423282d0b046d5b3/C in TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:18,565 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/81630c9d75e14440889a470c0b95bddd, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/06dc6270b31e498fad233379f8fb6091, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/5e16a187dc8b4957ae3aab370f24be74, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/62ed9585f56d49489011bb981df1cfb8] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp, totalSize=46.7 K 2023-04-27 03:54:18,567 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 81630c9d75e14440889a470c0b95bddd, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1682567654685 2023-04-27 03:54:18,567 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 06dc6270b31e498fad233379f8fb6091, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1682567654741 2023-04-27 03:54:18,568 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 5e16a187dc8b4957ae3aab370f24be74, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1682567655364 2023-04-27 03:54:18,568 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 62ed9585f56d49489011bb981df1cfb8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682567656524 2023-04-27 03:54:18,579 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): d66a88af1ba39e44423282d0b046d5b3#C#compaction#139 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:18,591 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/332f94de8bae488d8825be338483139a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/332f94de8bae488d8825be338483139a 2023-04-27 03:54:18,596 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d66a88af1ba39e44423282d0b046d5b3/C of d66a88af1ba39e44423282d0b046d5b3 into 332f94de8bae488d8825be338483139a(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:18,597 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:18,597 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., storeName=d66a88af1ba39e44423282d0b046d5b3/C, priority=12, startTime=1682567658102; duration=0sec 2023-04-27 03:54:18,597 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:18,675 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/fd82c02d0cc541b99055083636d38488 2023-04-27 03:54:18,680 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/f7ec5aa9b3bd41b796fd6f744f8d4d96 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/f7ec5aa9b3bd41b796fd6f744f8d4d96 2023-04-27 03:54:18,688 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/f7ec5aa9b3bd41b796fd6f744f8d4d96, entries=100, sequenceid=150, filesize=18.5 K 2023-04-27 03:54:18,689 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/cf41efa3cad64b1980e035e7a02056aa as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/cf41efa3cad64b1980e035e7a02056aa 2023-04-27 03:54:18,694 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/cf41efa3cad64b1980e035e7a02056aa, entries=100, sequenceid=150, filesize=9.4 K 2023-04-27 03:54:18,695 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/fd82c02d0cc541b99055083636d38488 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/fd82c02d0cc541b99055083636d38488 2023-04-27 03:54:18,701 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/fd82c02d0cc541b99055083636d38488, entries=100, sequenceid=150, filesize=9.4 K 2023-04-27 03:54:18,702 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for d66a88af1ba39e44423282d0b046d5b3 in 508ms, sequenceid=150, compaction requested=false 2023-04-27 03:54:18,702 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:18,702 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:18,702 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:18,702 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:18,702 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,702 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:18,702 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,704 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:18,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:18,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:18,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:18,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:18,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:18,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:18,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14c39a91[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:18,706 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:18,706 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,707 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,708 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:18,708 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:18,708 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:18,708 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,708 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:18,714 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:18,714 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:18,714 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:18,714 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:18,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:18,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:18,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:18,717 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:18,717 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:18,717 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:18,717 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:18,717 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:18,718 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,718 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,718 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:18,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:18,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:18,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,719 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:18,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:18,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:18,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:18,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:18,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:18,720 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:18,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:18,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,723 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,723 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:18,723 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:18,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:18,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:18,723 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:18,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:18,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:18,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:18,723 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:18,723 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:18,724 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,724 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,724 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:18,724 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:18,724 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,724 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:18,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:18,824 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:18,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:18,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:18,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:18,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:18,828 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:18,828 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:18,828 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:18,828 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:18,829 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,829 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:18,830 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,830 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:18,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:18,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:18,831 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,831 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:18,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:18,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:18,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:18,832 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:18,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:18,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:18,832 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:18,832 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:18,832 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:18,833 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:18,833 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:18,833 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:18,833 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:18,833 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:18,833 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:18,833 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:18,833 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:18,833 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,833 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:18,833 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:18,835 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,835 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:18,836 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,836 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:18,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:18,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15855446[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:18,838 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:18,838 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,839 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,839 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,839 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:18,839 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:18,839 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:18,839 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:18,841 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:18,841 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:18,841 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:18,841 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:18,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:18,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:18,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:18,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:18,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:18,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:18,845 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:18,845 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:18,845 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:18,845 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:18,845 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:18,847 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,847 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:18,847 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:18,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:18,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:18,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:18,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:18,847 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:18,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:18,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:18,849 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,852 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:18,852 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:18,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:18,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:18,852 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,853 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:18,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:18,853 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:18,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:18,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:18,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:18,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:18,853 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:18,853 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:18,853 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:18,853 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,853 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,853 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,853 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,953 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:18,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:18,954 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d66a88af1ba39e44423282d0b046d5b3 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:54:18,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:18,955 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=A 2023-04-27 03:54:18,956 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:18,956 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=B 2023-04-27 03:54:18,956 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:18,956 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=C 2023-04-27 03:54:18,956 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:18,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:18,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:18,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:18,959 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:18,959 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:18,959 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:18,959 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:18,961 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,961 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:18,962 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:18,962 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:18,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:18,962 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,962 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:18,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:18,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:18,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:18,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:18,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:18,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:18,966 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:18,966 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:18,966 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:18,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:18,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:18,968 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:18,968 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:18,968 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:18,968 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:18,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:18,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:18,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:18,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:18,970 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,970 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:18,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:18,970 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,970 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:18,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:18,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:18,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:18,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:18,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:18,973 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:18,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b82efcd[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:18,973 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,976 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,977 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:18,977 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:18,977 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:18,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:18,979 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:18,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:18,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:18,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:18,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:18,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:18,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:18,983 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:18,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:18,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:18,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:18,984 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:18,986 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,986 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,986 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:18,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:18,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:18,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:18,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:18,988 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:18,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:18,989 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/24e7bc7dd7504540913f4d5578b21e5c as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/24e7bc7dd7504540913f4d5578b21e5c 2023-04-27 03:54:18,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:18,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,992 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:18,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,995 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:18,995 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:18,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:18,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:18,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:18,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:18,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,996 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:18,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:18,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:18,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:18,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:18,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:18,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:18,996 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:18,996 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:18,997 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d66a88af1ba39e44423282d0b046d5b3/A of d66a88af1ba39e44423282d0b046d5b3 into 24e7bc7dd7504540913f4d5578b21e5c(size=25.5 K), total size for store is 44.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:18,997 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:18,997 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., storeName=d66a88af1ba39e44423282d0b046d5b3/A, priority=12, startTime=1682567658102; duration=0sec 2023-04-27 03:54:18,997 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:18,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:18,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:18,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567718994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:18,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567658981, max timestamp=1682567658996Num uniques 859; 2023-04-27 03:54:18,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:54:19,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567658981, max timestamp=1682567658996Num uniques 859; 2023-04-27 03:54:19,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567658981, max timestamp=1682567658996Num uniques 859; 2023-04-27 03:54:19,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:54:19,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:54:19,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:19,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52150 deadline: 1682567719077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:19,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567719079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:19,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567719082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:19,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567719083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,097 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:19,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:19,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:19,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:19,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567719100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:19,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:19,101 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:19,101 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:19,101 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:19,101 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:19,103 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,103 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:19,104 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,104 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:19,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:19,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:19,105 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,105 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:19,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:19,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:19,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:19,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:19,106 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:19,106 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:19,106 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:19,106 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:19,106 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:19,106 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:19,106 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:19,106 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:19,106 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:19,106 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:19,106 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:19,106 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:19,106 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:19,107 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,107 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:19,107 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:19,109 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,109 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:19,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:19,109 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,109 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:19,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:19,111 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b80dc96[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:19,111 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,112 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,112 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:19,112 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:19,112 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:19,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:19,114 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:19,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:19,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:19,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:19,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:19,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:19,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:19,117 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:19,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:19,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:19,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:19,117 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:19,122 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,122 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:19,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,122 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:19,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:19,125 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,126 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,129 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,129 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:19,129 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:19,129 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:19,129 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:19,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:19,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:19,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:19,129 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,129 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,129 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:19,129 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:19,129 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:19,129 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,129 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:19,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:19,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:19,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52150 deadline: 1682567719184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:19,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567719184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:19,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567719187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:19,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567719187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,229 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:19,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:19,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:19,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:19,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:19,233 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:19,233 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:19,233 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:19,233 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:19,233 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,233 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:19,234 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:19,235 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:19,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:19,235 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,235 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:19,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:19,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:19,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:19,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:19,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:19,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:19,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:19,236 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:19,236 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:19,236 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:19,237 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:19,237 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:19,237 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:19,237 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,237 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:19,237 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:19,238 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,238 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:19,238 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,239 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:19,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:19,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:19,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:19,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:19,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@231fcc65[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:19,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:19,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,242 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,242 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:19,242 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:19,242 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:19,242 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,242 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:19,244 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:19,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:19,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:19,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:19,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:19,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:19,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:19,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:19,246 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:19,246 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:19,246 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:19,246 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:19,247 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:19,248 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,248 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:19,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:19,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,248 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:19,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:19,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,250 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:19,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:19,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:19,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,251 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:19,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:19,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,253 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,253 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:19,253 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:19,253 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:19,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:19,253 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:19,253 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:19,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:19,253 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:19,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:19,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:19,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:19,254 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,254 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:19,254 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,254 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,254 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:19,303 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:19,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567719302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,354 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:19,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:19,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:19,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:19,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:19,358 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:19,358 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:19,358 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:19,358 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:19,359 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,359 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:19,361 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,361 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:19,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:19,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:19,361 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,361 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:19,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:19,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:19,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:19,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:19,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:19,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:19,363 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:19,363 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:19,363 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:19,363 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:19,363 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:19,363 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:19,363 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:19,363 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:19,363 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:19,363 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:19,363 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:19,363 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,363 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:19,363 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:19,365 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,365 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:19,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:19,365 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,365 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:19,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:19,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:19,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:19,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:19,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:19,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c078b24[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:19,367 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:19,367 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,368 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,369 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:19,369 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:19,369 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:19,369 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,369 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:19,370 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,370 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:19,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:19,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:19,370 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:19,370 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:19,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:19,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:19,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:19,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:19,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:19,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:19,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:19,373 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:19,373 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:19,373 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:19,373 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:19,373 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:19,375 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,375 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:19,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,375 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:19,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:19,376 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:19,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:19,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:19,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:19,377 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:19,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:19,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:19,380 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:19,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:19,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:19,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:19,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:19,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:19,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:19,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:19,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,380 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:19,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,380 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:19,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:19,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:19,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:19,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52150 deadline: 1682567719387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:19,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567719387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:19,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567719390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:19,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567719392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,409 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427876af8d6a40a40d18db36ff6d094e904 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427876af8d6a40a40d18db36ff6d094e904 2023-04-27 03:54:19,416 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=164, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/dac46cdff3bf4472a323d30a772d237f 2023-04-27 03:54:19,432 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/b4ac997bec3d4296938ffbb05da5bb4c 2023-04-27 03:54:19,454 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/f164d30330584b3783fc2ae838d0fbe2 2023-04-27 03:54:19,460 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/dac46cdff3bf4472a323d30a772d237f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/dac46cdff3bf4472a323d30a772d237f 2023-04-27 03:54:19,464 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/dac46cdff3bf4472a323d30a772d237f, entries=400, sequenceid=164, filesize=59.4 K 2023-04-27 03:54:19,464 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/b4ac997bec3d4296938ffbb05da5bb4c as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/b4ac997bec3d4296938ffbb05da5bb4c 2023-04-27 03:54:19,468 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/b4ac997bec3d4296938ffbb05da5bb4c, entries=150, sequenceid=164, filesize=11.8 K 2023-04-27 03:54:19,469 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/f164d30330584b3783fc2ae838d0fbe2 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/f164d30330584b3783fc2ae838d0fbe2 2023-04-27 03:54:19,473 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/f164d30330584b3783fc2ae838d0fbe2, entries=150, sequenceid=164, filesize=11.8 K 2023-04-27 03:54:19,473 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for d66a88af1ba39e44423282d0b046d5b3 in 519ms, sequenceid=164, compaction requested=true 2023-04-27 03:54:19,473 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:19,474 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:19,474 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:19,474 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:19,474 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:19,474 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:19,475 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33954 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:19,475 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): d66a88af1ba39e44423282d0b046d5b3/B is initiating minor compaction (all files) 2023-04-27 03:54:19,475 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d66a88af1ba39e44423282d0b046d5b3/B in TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:19,475 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/88753208d5d74c6eba0522e2948b3a99, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/cf41efa3cad64b1980e035e7a02056aa, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/b4ac997bec3d4296938ffbb05da5bb4c] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp, totalSize=33.2 K 2023-04-27 03:54:19,475 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 105888 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:19,476 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): d66a88af1ba39e44423282d0b046d5b3/A is initiating minor compaction (all files) 2023-04-27 03:54:19,476 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d66a88af1ba39e44423282d0b046d5b3/A in TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:19,476 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/24e7bc7dd7504540913f4d5578b21e5c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/f7ec5aa9b3bd41b796fd6f744f8d4d96, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/dac46cdff3bf4472a323d30a772d237f] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp, totalSize=103.4 K 2023-04-27 03:54:19,476 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 88753208d5d74c6eba0522e2948b3a99, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682567656524 2023-04-27 03:54:19,476 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 24e7bc7dd7504540913f4d5578b21e5c, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682567656524 2023-04-27 03:54:19,477 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting cf41efa3cad64b1980e035e7a02056aa, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1682567656840 2023-04-27 03:54:19,477 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting f7ec5aa9b3bd41b796fd6f744f8d4d96, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1682567656840 2023-04-27 03:54:19,477 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting b4ac997bec3d4296938ffbb05da5bb4c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1682567658946 2023-04-27 03:54:19,477 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting dac46cdff3bf4472a323d30a772d237f, keycount=400, bloomtype=ROW, size=59.4 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1682567658916 2023-04-27 03:54:19,481 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:19,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:19,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:19,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:19,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:19,486 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:19,486 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:19,486 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:19,486 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:19,486 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,486 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:19,488 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:19,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:19,488 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:19,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:19,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:19,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:19,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:19,491 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): d66a88af1ba39e44423282d0b046d5b3#B#compaction#144 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:19,492 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:19,492 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:19,492 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:19,492 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:19,492 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:19,492 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:19,492 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:19,492 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:19,492 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:19,492 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,493 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:19,492 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d66a88af1ba39e44423282d0b046d5b3 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-27 03:54:19,493 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=A 2023-04-27 03:54:19,493 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:19,493 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=B 2023-04-27 03:54:19,493 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:19,493 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=C 2023-04-27 03:54:19,493 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:19,497 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): d66a88af1ba39e44423282d0b046d5b3#A#compaction#143 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:19,521 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427499e79e45d5444668200597f744ddbff to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427499e79e45d5444668200597f744ddbff 2023-04-27 03:54:19,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:19,608 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:19,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:19,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567719630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:19,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567719690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:19,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567719692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:19,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52150 deadline: 1682567719692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:19,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567719696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:19,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567719736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,904 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/6cdd4a851aa048d683cbb9b38d2e9ea2 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/6cdd4a851aa048d683cbb9b38d2e9ea2 2023-04-27 03:54:19,909 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d66a88af1ba39e44423282d0b046d5b3/B of d66a88af1ba39e44423282d0b046d5b3 into 6cdd4a851aa048d683cbb9b38d2e9ea2(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:19,909 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:19,909 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., storeName=d66a88af1ba39e44423282d0b046d5b3/B, priority=13, startTime=1682567659474; duration=0sec 2023-04-27 03:54:19,909 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:19,909 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:19,910 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33954 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:19,911 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): d66a88af1ba39e44423282d0b046d5b3/C is initiating minor compaction (all files) 2023-04-27 03:54:19,911 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d66a88af1ba39e44423282d0b046d5b3/C in TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:19,911 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/332f94de8bae488d8825be338483139a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/fd82c02d0cc541b99055083636d38488, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/f164d30330584b3783fc2ae838d0fbe2] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp, totalSize=33.2 K 2023-04-27 03:54:19,911 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 332f94de8bae488d8825be338483139a, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682567656524 2023-04-27 03:54:19,912 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting fd82c02d0cc541b99055083636d38488, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1682567656840 2023-04-27 03:54:19,912 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting f164d30330584b3783fc2ae838d0fbe2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1682567658946 2023-04-27 03:54:19,919 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): d66a88af1ba39e44423282d0b046d5b3#C#compaction#146 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:19,926 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=200, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/6844eb8c523c464db275b12d9bcecd3a 2023-04-27 03:54:19,930 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/6602ced3ae5340a4bcba65e865b26084 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/6602ced3ae5340a4bcba65e865b26084 2023-04-27 03:54:19,940 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d66a88af1ba39e44423282d0b046d5b3/A of d66a88af1ba39e44423282d0b046d5b3 into 6602ced3ae5340a4bcba65e865b26084(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:19,940 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:19,940 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., storeName=d66a88af1ba39e44423282d0b046d5b3/A, priority=13, startTime=1682567659474; duration=0sec 2023-04-27 03:54:19,940 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:19,943 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/3629b732a6554761a9460f960db3cd2a 2023-04-27 03:54:19,943 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:19,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567719941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,956 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/e82ded12040f4f668a30eaa63e137fe8 2023-04-27 03:54:19,968 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/6844eb8c523c464db275b12d9bcecd3a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/6844eb8c523c464db275b12d9bcecd3a 2023-04-27 03:54:19,973 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/6844eb8c523c464db275b12d9bcecd3a, entries=150, sequenceid=200, filesize=25.3 K 2023-04-27 03:54:19,974 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/3629b732a6554761a9460f960db3cd2a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/3629b732a6554761a9460f960db3cd2a 2023-04-27 03:54:19,978 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/3629b732a6554761a9460f960db3cd2a, entries=150, sequenceid=200, filesize=11.8 K 2023-04-27 03:54:19,979 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/e82ded12040f4f668a30eaa63e137fe8 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/e82ded12040f4f668a30eaa63e137fe8 2023-04-27 03:54:19,983 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/e82ded12040f4f668a30eaa63e137fe8, entries=150, sequenceid=200, filesize=11.8 K 2023-04-27 03:54:19,985 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for d66a88af1ba39e44423282d0b046d5b3 in 492ms, sequenceid=200, compaction requested=false 2023-04-27 03:54:19,985 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:19,985 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:19,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:19,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:19,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:19,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:19,986 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,987 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:19,987 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,987 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:19,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:19,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:19,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:19,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6007f3ee[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:19,990 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:19,990 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,992 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:19,992 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:19,992 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:19,992 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,992 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:19,992 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:19,993 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:19,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:19,994 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:19,994 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:19,994 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:19,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:19,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:19,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:19,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:19,996 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:19,996 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:19,996 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:19,996 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:19,997 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:19,998 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,998 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,998 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:19,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:19,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:19,999 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,000 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:20,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,007 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,007 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,007 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,007 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:20,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:20,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:20,007 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:20,007 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,007 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:20,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:20,008 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:20,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:20,008 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,008 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:20,008 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,008 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:20,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:20,108 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:20,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:20,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:20,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:20,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:20,111 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:20,111 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:20,112 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:20,112 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:20,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,114 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:20,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:20,114 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,115 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,115 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:20,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:20,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:20,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:20,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:20,116 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:20,116 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:20,116 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:20,116 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:20,116 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:20,116 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:20,116 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:20,116 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d66a88af1ba39e44423282d0b046d5b3 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-27 03:54:20,116 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=A 2023-04-27 03:54:20,116 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:20,116 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=B 2023-04-27 03:54:20,116 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:20,116 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=C 2023-04-27 03:54:20,117 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:20,166 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427878846bc885545debb1f5843845fdcec to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427878846bc885545debb1f5843845fdcec 2023-04-27 03:54:20,183 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=213, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/76f1befaa3374353924e07230d307eee 2023-04-27 03:54:20,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:20,198 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:20,222 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/ca39d6000e1f48509d96e37057d90dcf 2023-04-27 03:54:20,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:20,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52150 deadline: 1682567720219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567660217, max timestamp=1682567660222Num uniques 859; 2023-04-27 03:54:20,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:54:20,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567660217, max timestamp=1682567660222Num uniques 859; 2023-04-27 03:54:20,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:54:20,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=183, cellSize=8378, totalHeapSize=19272, min timestamp=1682567660217, max timestamp=1682567660222Num uniques 859; 2023-04-27 03:54:20,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=8378, suffix heap size=117064,new segment heap size=19272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=183 2023-04-27 03:54:20,242 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/91986859bcfc46e68ffccaaeec4c6b60 2023-04-27 03:54:20,248 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/76f1befaa3374353924e07230d307eee as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/76f1befaa3374353924e07230d307eee 2023-04-27 03:54:20,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:20,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567720248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:20,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567720249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,253 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/76f1befaa3374353924e07230d307eee, entries=150, sequenceid=213, filesize=25.3 K 2023-04-27 03:54:20,254 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/ca39d6000e1f48509d96e37057d90dcf as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/ca39d6000e1f48509d96e37057d90dcf 2023-04-27 03:54:20,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:20,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567720251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:20,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567720251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,258 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/ca39d6000e1f48509d96e37057d90dcf, entries=150, sequenceid=213, filesize=11.8 K 2023-04-27 03:54:20,259 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/91986859bcfc46e68ffccaaeec4c6b60 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/91986859bcfc46e68ffccaaeec4c6b60 2023-04-27 03:54:20,263 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/91986859bcfc46e68ffccaaeec4c6b60, entries=150, sequenceid=213, filesize=11.8 K 2023-04-27 03:54:20,264 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=126.31 KB/129344 for d66a88af1ba39e44423282d0b046d5b3 in 148ms, sequenceid=213, compaction requested=true 2023-04-27 03:54:20,264 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:20,264 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:20,264 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:20,264 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:20,264 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,264 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:20,264 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:20,266 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,266 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:20,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:20,266 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,266 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:20,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:20,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:20,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:20,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:20,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dcee714[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:20,268 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:20,268 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,270 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,270 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:20,270 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:20,270 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:20,270 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,270 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:20,272 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,272 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:20,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:20,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:20,272 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:20,272 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:20,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:20,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:20,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:20,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:20,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:20,274 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:20,274 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:20,274 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:20,274 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:20,274 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:20,275 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,275 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,276 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:20,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:20,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:20,276 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,277 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,279 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:20,279 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:20,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:20,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:20,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:20,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:20,279 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,279 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:20,280 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,280 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,280 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:20,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:20,280 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:20,280 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:20,280 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,280 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,280 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:20,329 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d66a88af1ba39e44423282d0b046d5b3 3/3 column families, dataSize=133.02 KB heapSize=377.66 KB 2023-04-27 03:54:20,329 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=A 2023-04-27 03:54:20,329 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:20,329 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=B 2023-04-27 03:54:20,329 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:20,329 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=C 2023-04-27 03:54:20,329 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:20,330 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/0f7229f465bf43a5ba2a1bcc3c887829 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/0f7229f465bf43a5ba2a1bcc3c887829 2023-04-27 03:54:20,337 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d66a88af1ba39e44423282d0b046d5b3/C of d66a88af1ba39e44423282d0b046d5b3 into 0f7229f465bf43a5ba2a1bcc3c887829(size=12.1 K), total size for store is 35.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:20,337 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:20,337 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., storeName=d66a88af1ba39e44423282d0b046d5b3/C, priority=13, startTime=1682567659474; duration=0sec 2023-04-27 03:54:20,337 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:20,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:20,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567720352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:20,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52150 deadline: 1682567720354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:20,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567720354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,361 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:20,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567720359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,361 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:20,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567720359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,380 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:20,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:20,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:20,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:20,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:20,383 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:20,383 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:20,383 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:20,383 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:20,384 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,384 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,386 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,386 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:20,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:20,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:20,386 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,386 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:20,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:20,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:20,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:20,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:20,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:20,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:20,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:20,388 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:20,388 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:20,388 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:20,388 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:20,388 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:20,388 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:20,388 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:20,388 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:20,388 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,388 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:20,388 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:20,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,389 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:20,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:20,390 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,390 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:20,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:20,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:20,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:20,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:20,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@106f762c[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:20,391 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:20,392 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,393 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,393 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,393 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:20,393 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:20,393 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:20,393 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:20,395 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,395 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:20,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:20,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:20,395 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:20,395 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:20,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:20,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:20,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:20,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:20,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:20,397 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:20,397 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:20,397 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:20,397 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:20,397 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:20,400 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,400 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:20,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:20,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,400 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:20,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:20,400 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:20,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:20,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:20,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,401 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:20,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,403 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,403 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:20,403 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:20,403 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:20,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:20,403 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:20,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:20,403 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,403 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:20,404 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:20,404 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:20,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:20,404 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,404 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,404 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:20,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:20,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:20,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52150 deadline: 1682567720458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:20,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567720458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,504 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:20,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:20,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:20,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:20,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:20,507 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:20,507 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:20,507 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:20,507 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:20,508 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,508 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,509 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,509 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:20,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:20,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:20,509 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,509 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:20,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,510 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:20,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:20,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:20,510 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:20,510 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:20,510 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:20,511 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:20,511 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:20,511 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:20,511 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:20,511 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:20,511 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:20,511 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:20,511 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:20,511 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,511 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:20,511 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:20,512 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,512 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:20,513 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,513 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:20,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:20,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@301191ae[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:20,514 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:20,514 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,517 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,517 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:20,517 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:20,517 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:20,517 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,517 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:20,518 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,518 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:20,518 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:20,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:20,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:20,519 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:20,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:20,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:20,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:20,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:20,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:20,521 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:20,521 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:20,521 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:20,521 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:20,521 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:20,523 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:20,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:20,523 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,523 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:20,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:20,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:20,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:20,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:20,525 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:20,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,526 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,538 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:20,538 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,538 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:20,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:20,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:20,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:20,538 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:20,538 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:20,538 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,538 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,538 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:20,538 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,538 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:20,538 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,538 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:20,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:20,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567720556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:20,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:20,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567720562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567720563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,638 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:20,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:20,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:20,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:20,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:20,641 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:20,642 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:20,642 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:20,642 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:20,642 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,642 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,643 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,643 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:20,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:20,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:20,643 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,643 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:20,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:20,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:20,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:20,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:20,644 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:20,644 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:20,645 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:20,645 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:20,645 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:20,645 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:20,645 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:20,645 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:20,645 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:20,645 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:20,645 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:20,645 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,645 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:20,645 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:20,647 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,647 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:20,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:20,648 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,648 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:20,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:20,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:20,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:20,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:20,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f569cc1[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:20,649 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:20,649 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,651 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,651 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:20,651 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:20,651 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:20,651 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,652 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:20,653 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:20,653 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:20,653 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,653 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:20,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:20,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:20,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:20,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:20,655 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:20,655 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:20,655 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:20,655 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:20,655 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:20,657 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,657 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:20,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:20,657 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:20,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:20,658 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:20,659 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:20,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,661 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,661 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,661 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,662 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:20,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:20,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:20,662 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:20,662 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:20,662 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,662 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,662 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,662 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:20,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:20,662 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,662 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:20,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:20,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:20,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:20,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52150 deadline: 1682567720665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:20,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567720665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,749 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304279b66090538b64aa894eef2407d2b9d4c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304279b66090538b64aa894eef2407d2b9d4c 2023-04-27 03:54:20,762 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:20,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:20,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:20,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:20,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:20,765 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:20,765 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:20,765 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:20,765 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:20,766 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,766 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,768 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:20,768 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:20,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:20,769 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,769 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,769 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:20,770 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:20,770 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:20,770 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:20,770 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:20,770 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:20,770 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:20,770 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:20,770 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:20,770 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:20,770 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:20,770 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:20,770 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:20,770 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:20,770 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,770 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:20,771 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:20,772 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,773 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:20,773 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,773 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:20,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:20,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:20,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@559a2078[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:20,775 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:20,775 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,776 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,776 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:20,776 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:20,776 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:20,776 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,776 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:20,777 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:20,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:20,777 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:20,778 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:20,778 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:20,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:20,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:20,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:20,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:20,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:20,781 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:20,781 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:20,781 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:20,781 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:20,781 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:20,783 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,783 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,783 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:20,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:20,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:20,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:20,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:20,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:20,784 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:20,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,785 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,788 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:20,788 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:20,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:20,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:20,788 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,788 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,788 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,788 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,788 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:20,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:20,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:20,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:20,788 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,789 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:20,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:20,789 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:20,789 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,789 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:20,864 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:20,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567720861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:20,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567720868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:20,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567720868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,889 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:20,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:20,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:20,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:20,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:20,893 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:20,893 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:20,893 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:20,893 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:20,893 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,893 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,894 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:20,894 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:20,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:20,895 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,895 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:20,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:20,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:20,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:20,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:20,896 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:20,896 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:20,896 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:20,896 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:20,896 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:20,896 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:20,896 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:20,896 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:20,896 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:20,896 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:20,896 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:20,896 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,897 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:20,897 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:20,898 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,898 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:20,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:20,898 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,898 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:20,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:20,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:20,900 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:20,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3eafe2f1[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:20,900 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,902 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,902 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:20,902 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:20,902 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,902 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:20,902 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:20,904 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,904 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:20,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:20,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:20,904 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:20,904 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:20,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:20,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:20,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:20,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:20,907 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:20,908 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:20,908 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:20,908 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:20,908 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:20,909 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,909 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:20,909 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:20,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:20,910 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,911 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,913 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,913 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,913 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:20,913 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:20,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:20,913 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,913 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:20,913 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:20,913 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:20,913 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:20,913 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:20,913 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:20,913 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:20,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:20,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:20,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:20,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:20,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567720971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:20,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:20,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52150 deadline: 1682567720971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,013 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:21,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:21,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:21,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:21,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:21,017 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:21,017 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:21,017 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:21,017 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:21,017 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,017 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,019 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:21,019 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:21,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:21,020 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,020 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:21,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:21,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:21,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:21,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:21,021 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:21,021 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:21,021 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:21,021 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:21,021 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:21,021 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:21,021 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:21,021 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:21,021 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:21,021 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:21,021 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:21,021 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,021 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:21,021 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:21,029 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,029 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:21,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:21,030 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,030 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:21,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:21,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:21,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:21,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20f886b[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:21,032 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:21,032 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,033 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,033 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:21,033 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:21,033 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:21,033 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,033 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:21,034 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:21,034 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:21,034 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,034 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:21,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:21,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:21,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:21,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:21,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:21,037 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:21,037 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:21,037 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:21,037 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:21,037 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:21,040 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,040 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:21,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,040 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:21,040 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:21,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:21,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,041 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,044 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:21,044 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,044 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:21,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:21,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:21,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:21,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:21,044 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,044 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,044 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,044 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,044 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:21,044 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:21,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:21,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:21,044 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:21,044 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:21,044 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,144 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:21,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:21,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:21,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:21,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:21,148 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:21,148 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:21,148 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:21,148 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:21,148 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,148 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,149 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,149 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:21,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:21,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:21,150 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,150 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:21,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:21,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:21,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:21,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:21,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:21,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:21,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:21,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:21,151 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:21,151 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:21,151 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:21,151 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:21,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:21,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:21,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:21,152 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:21,153 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,153 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:21,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:21,154 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:21,154 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:21,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:21,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:21,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:21,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5964bc7f[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:21,155 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:21,155 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,157 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=249, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/a02ec880ad66426c952cc31f946d26b8 2023-04-27 03:54:21,157 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,158 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:21,158 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:21,158 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,158 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:21,158 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:21,159 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:21,159 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,159 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:21,159 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:21,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:21,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:21,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:21,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:21,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:21,163 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:21,163 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:21,163 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:21,163 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:21,163 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:21,165 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:21,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:21,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:21,165 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:21,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,166 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:21,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,168 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,168 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:21,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,169 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,169 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:21,169 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:21,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:21,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:21,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:21,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:21,169 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,169 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:21,169 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,169 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,169 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,169 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:21,169 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:21,172 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:21,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:21,272 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:21,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:21,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:21,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:21,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:21,275 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:21,275 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:21,275 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:21,275 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:21,276 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,276 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,278 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:21,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:21,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:21,278 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,279 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,279 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:21,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:21,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,279 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:21,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:21,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:21,280 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:21,280 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:21,280 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:21,281 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:21,281 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:21,281 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:21,281 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:21,281 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:21,281 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:21,281 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:21,281 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:21,281 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,281 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:21,281 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:21,283 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,283 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:21,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:21,283 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,283 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:21,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:21,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:21,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:21,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:21,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d79cecb[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:21,285 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:21,285 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,286 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,286 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:21,286 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:21,286 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:21,286 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,286 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:21,287 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,287 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:21,287 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:21,287 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:21,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:21,290 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:21,290 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:21,290 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:21,290 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:21,290 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:21,292 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,292 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:21,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,292 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:21,293 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,294 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,296 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,296 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,296 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,296 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:21,296 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:21,296 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:21,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:21,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:21,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:21,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:21,296 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:21,297 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,297 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,297 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,297 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:21,297 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:21,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:21,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:21,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:21,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567721368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:21,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567721370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:21,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567721371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,397 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:21,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:21,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:21,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:21,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:21,401 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:21,401 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:21,401 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:21,401 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:21,401 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,401 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,402 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:21,402 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:21,403 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,403 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:21,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:21,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:21,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:21,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:21,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:21,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:21,405 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:21,405 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:21,405 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:21,405 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:21,405 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:21,405 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:21,405 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:21,405 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:21,405 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,405 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:21,405 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:21,406 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,406 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:21,407 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,407 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:21,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:21,408 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:21,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c0007ff[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:21,408 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,412 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,413 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,413 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:21,413 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:21,413 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:21,413 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:21,414 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:21,414 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,414 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:21,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,414 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:21,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:21,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:21,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:21,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:21,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:21,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:21,416 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:21,416 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:21,417 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:21,417 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:21,417 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:21,418 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,418 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:21,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,418 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:21,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:21,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:21,419 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,420 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:21,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,422 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,422 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,422 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:21,422 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:21,422 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:21,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:21,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:21,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:21,423 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,422 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:21,423 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:21,423 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:21,423 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,423 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:21,423 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:21,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:21,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:21,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52150 deadline: 1682567721475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:21,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567721478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,523 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:21,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:21,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:21,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:21,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:21,526 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:21,527 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:21,527 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:21,527 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:21,527 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,527 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,530 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:21,530 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:21,530 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,530 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:21,531 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:21,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:21,531 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:21,531 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:21,531 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:21,532 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:21,532 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:21,532 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:21,532 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:21,532 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:21,532 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:21,532 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:21,532 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:21,532 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,532 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:21,532 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:21,534 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,534 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:21,535 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,535 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:21,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:21,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62b298c4[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:21,537 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:21,537 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,538 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,538 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:21,538 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:21,538 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:21,538 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,538 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:21,539 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,539 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:21,540 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:21,540 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:21,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:21,542 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:21,542 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:21,542 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:21,542 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:21,542 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:21,545 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,545 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:21,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:21,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,545 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:21,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:21,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:21,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:21,545 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:21,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,547 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:21,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,549 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,549 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:21,549 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,549 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,549 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:21,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:21,549 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:21,550 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:21,550 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:21,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:21,549 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:21,549 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,550 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,550 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:21,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:21,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:21,572 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/d356c57c6d81449f98be5bdd4efb71e3 2023-04-27 03:54:21,650 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:21,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:21,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:21,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:21,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:21,654 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:21,655 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:21,655 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:21,655 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:21,655 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,655 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,657 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:21,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:21,657 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:21,658 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,658 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:21,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:21,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:21,658 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:21,659 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:21,659 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:21,664 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:21,664 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:21,664 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:21,664 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:21,664 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:21,664 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:21,665 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:21,665 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:21,665 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,665 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:21,665 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:21,667 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,667 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:21,667 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,667 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:21,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:21,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e4947a9[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:21,669 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:21,669 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,671 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,671 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:21,671 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:21,671 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:21,673 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,673 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:21,673 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:21,673 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:21,673 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:21,673 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:21,673 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:21,673 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:21,673 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:21,673 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:21,673 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:21,675 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,676 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,676 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:21,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:21,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,676 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:21,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:21,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:21,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:21,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,678 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:21,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:21,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,680 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:21,680 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:21,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:21,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:21,681 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,681 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,681 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,681 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,681 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,681 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,681 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:21,681 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:21,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:21,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:21,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:21,681 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:21,682 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:21,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:21,782 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:21,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:21,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:21,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:21,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:21,787 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:21,787 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:21,787 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:21,787 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:21,787 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,787 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,794 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:21,794 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,795 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,795 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:21,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:21,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:21,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:21,796 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:21,796 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:21,796 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:21,796 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:21,798 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,798 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:21,799 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:21,799 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:21,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:21,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d55cd3d[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:21,800 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:21,800 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,802 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,802 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:21,802 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:21,802 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:21,802 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,802 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:21,803 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:21,803 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:21,803 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:21,804 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:21,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:21,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:21,807 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:21,807 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:21,808 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:21,808 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:21,808 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:21,812 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,812 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,813 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:21,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:21,814 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,815 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,818 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,818 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,818 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,818 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,818 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,818 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:21,818 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:21,818 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:21,818 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:21,818 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:21,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:21,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:21,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:21,819 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:21,819 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:21,919 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:21,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:21,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:21,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:21,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:21,923 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:21,923 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:21,923 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:21,923 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:21,923 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,923 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,925 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:21,925 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,925 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,925 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:21,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:21,926 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:21,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:21,927 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:21,927 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:21,927 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:21,927 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:21,927 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:21,927 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:21,927 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:21,927 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:21,927 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:21,927 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:21,927 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:21,927 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,927 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:21,927 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:21,929 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,929 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:21,929 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,929 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:21,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:21,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11761c97[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:21,931 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:21,932 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,934 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,934 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:21,934 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,934 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:21,934 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:21,934 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:21,936 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:21,936 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:21,936 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,936 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:21,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:21,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:21,938 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:21,938 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:21,939 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:21,939 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:21,939 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:21,941 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,941 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:21,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,941 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:21,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:21,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:21,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:21,942 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:21,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,943 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:21,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:21,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,945 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:21,945 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:21,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:21,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:21,945 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,946 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,946 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:21,946 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:21,946 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,946 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:21,946 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:21,946 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:21,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:21,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:21,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:21,946 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:21,946 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:21,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:21,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:21,984 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=43.57 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/52961e2ca07c4ce9adcd71a9d01283e5 2023-04-27 03:54:21,988 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/a02ec880ad66426c952cc31f946d26b8 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/a02ec880ad66426c952cc31f946d26b8 2023-04-27 03:54:21,993 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/a02ec880ad66426c952cc31f946d26b8, entries=200, sequenceid=249, filesize=32.1 K 2023-04-27 03:54:21,994 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/d356c57c6d81449f98be5bdd4efb71e3 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/d356c57c6d81449f98be5bdd4efb71e3 2023-04-27 03:54:21,998 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/d356c57c6d81449f98be5bdd4efb71e3, entries=150, sequenceid=249, filesize=11.8 K 2023-04-27 03:54:21,999 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/52961e2ca07c4ce9adcd71a9d01283e5 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/52961e2ca07c4ce9adcd71a9d01283e5 2023-04-27 03:54:22,006 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/52961e2ca07c4ce9adcd71a9d01283e5, entries=150, sequenceid=249, filesize=11.8 K 2023-04-27 03:54:22,007 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~133.02 KB/136214, heapSize ~377.61 KB/386672, currentSize=46.96 KB/48090 for d66a88af1ba39e44423282d0b046d5b3 in 1678ms, sequenceid=249, compaction requested=true 2023-04-27 03:54:22,007 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:22,007 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:22,007 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 03:54:22,008 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 03:54:22,007 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:22,008 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:22,009 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110916 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 03:54:22,009 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): d66a88af1ba39e44423282d0b046d5b3/A is initiating minor compaction (all files) 2023-04-27 03:54:22,009 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d66a88af1ba39e44423282d0b046d5b3/A in TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:22,009 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/6602ced3ae5340a4bcba65e865b26084, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/6844eb8c523c464db275b12d9bcecd3a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/76f1befaa3374353924e07230d307eee, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/a02ec880ad66426c952cc31f946d26b8] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp, totalSize=108.3 K 2023-04-27 03:54:22,009 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48486 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 03:54:22,009 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): d66a88af1ba39e44423282d0b046d5b3/B is initiating minor compaction (all files) 2023-04-27 03:54:22,010 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d66a88af1ba39e44423282d0b046d5b3/B in TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:22,010 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/6cdd4a851aa048d683cbb9b38d2e9ea2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/3629b732a6554761a9460f960db3cd2a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/ca39d6000e1f48509d96e37057d90dcf, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/d356c57c6d81449f98be5bdd4efb71e3] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp, totalSize=47.3 K 2023-04-27 03:54:22,011 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 6602ced3ae5340a4bcba65e865b26084, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1682567658946 2023-04-27 03:54:22,011 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 6cdd4a851aa048d683cbb9b38d2e9ea2, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1682567658946 2023-04-27 03:54:22,011 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 6844eb8c523c464db275b12d9bcecd3a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1682567659077 2023-04-27 03:54:22,011 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 3629b732a6554761a9460f960db3cd2a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1682567659077 2023-04-27 03:54:22,016 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 76f1befaa3374353924e07230d307eee, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682567659613 2023-04-27 03:54:22,016 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting ca39d6000e1f48509d96e37057d90dcf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682567659613 2023-04-27 03:54:22,017 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting a02ec880ad66426c952cc31f946d26b8, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1682567660242 2023-04-27 03:54:22,017 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting d356c57c6d81449f98be5bdd4efb71e3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1682567660242 2023-04-27 03:54:22,033 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): d66a88af1ba39e44423282d0b046d5b3#B#compaction#156 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:22,046 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:22,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:22,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:22,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:22,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:22,052 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:22,052 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:22,052 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:22,053 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:22,053 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,053 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,053 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): d66a88af1ba39e44423282d0b046d5b3#A#compaction#155 average throughput is 0.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:22,056 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:22,056 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:22,056 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,056 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:22,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:22,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:22,088 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:22,088 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:22,088 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:22,088 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:22,088 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:22,088 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:22,089 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:22,089 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d66a88af1ba39e44423282d0b046d5b3 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:54:22,089 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=A 2023-04-27 03:54:22,089 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:22,089 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=B 2023-04-27 03:54:22,089 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:22,089 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=C 2023-04-27 03:54:22,090 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:22,131 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304272244e81f8cf34dcfab2652a523313ee2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272244e81f8cf34dcfab2652a523313ee2 2023-04-27 03:54:22,148 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=260, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/5f0a6951ac814b769bd1ec7f172f7dbd 2023-04-27 03:54:22,170 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/4479c57b5ff542ca904782887d659452 2023-04-27 03:54:22,201 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/251d34b133d34d6b923d7c40c0dbd153 2023-04-27 03:54:22,207 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/5f0a6951ac814b769bd1ec7f172f7dbd as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/5f0a6951ac814b769bd1ec7f172f7dbd 2023-04-27 03:54:22,212 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/5f0a6951ac814b769bd1ec7f172f7dbd, entries=100, sequenceid=260, filesize=18.6 K 2023-04-27 03:54:22,213 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/4479c57b5ff542ca904782887d659452 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/4479c57b5ff542ca904782887d659452 2023-04-27 03:54:22,219 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/4479c57b5ff542ca904782887d659452, entries=100, sequenceid=260, filesize=9.5 K 2023-04-27 03:54:22,220 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/251d34b133d34d6b923d7c40c0dbd153 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/251d34b133d34d6b923d7c40c0dbd153 2023-04-27 03:54:22,224 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/251d34b133d34d6b923d7c40c0dbd153, entries=100, sequenceid=260, filesize=9.5 K 2023-04-27 03:54:22,225 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for d66a88af1ba39e44423282d0b046d5b3 in 136ms, sequenceid=260, compaction requested=true 2023-04-27 03:54:22,226 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:22,226 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:22,226 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:22,226 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:22,226 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,226 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:22,226 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:22,227 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,227 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:22,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:22,228 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,228 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:22,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:22,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:22,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:22,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:22,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6774c2c5[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:22,230 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:22,230 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,231 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:22,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:22,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:22,231 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,232 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:22,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:22,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:22,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:22,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:22,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:22,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:22,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:22,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:22,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:22,236 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:22,236 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:22,236 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:22,236 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:22,236 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:22,239 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,239 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:22,239 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:22,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,244 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:22,244 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,244 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,244 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,244 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:22,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:22,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:22,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:22,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:22,244 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:22,245 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:22,245 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:22,244 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,245 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,245 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,245 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:22,345 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:22,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:22,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:22,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:22,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:22,349 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:22,349 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:22,350 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:22,350 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:22,350 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,350 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,351 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:22,351 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:22,352 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,352 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,353 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:22,353 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:22,353 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:22,353 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:22,353 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:22,353 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:22,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:22,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:22,354 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:22,354 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:22,354 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:22,354 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:22,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:22,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:22,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:22,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:22,356 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,356 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:22,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:22,356 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,356 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:22,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:22,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:22,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:22,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:22,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33d4a58d[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:22,358 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:22,358 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,360 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:22,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:22,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:22,361 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,361 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:22,362 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:22,362 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:22,362 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,362 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:22,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:22,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:22,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:22,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:22,365 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:22,365 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:22,365 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:22,365 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:22,365 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:22,367 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,367 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:22,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,367 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:22,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:22,368 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:22,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:22,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,370 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:22,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,376 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,376 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,376 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,376 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,376 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:22,376 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:22,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:22,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:22,376 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,377 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,377 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:22,377 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:22,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:22,376 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:22,377 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:22,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:22,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:22,415 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d66a88af1ba39e44423282d0b046d5b3 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:54:22,415 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=A 2023-04-27 03:54:22,415 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:22,415 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=B 2023-04-27 03:54:22,415 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:22,415 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=C 2023-04-27 03:54:22,415 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:22,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567662434, max timestamp=1682567662454Num uniques 859; 2023-04-27 03:54:22,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:54:22,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567662434, max timestamp=1682567662454Num uniques 859; 2023-04-27 03:54:22,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567662434, max timestamp=1682567662454Num uniques 859; 2023-04-27 03:54:22,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:54:22,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:54:22,477 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:22,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:22,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:22,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:22,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:22,480 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:22,480 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:22,480 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:22,480 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:22,481 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,481 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,482 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:22,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:22,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:22,483 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,483 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,483 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:22,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:22,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567722479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:22,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52150 deadline: 1682567722480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:22,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:22,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:22,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:22,484 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:22,484 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:22,484 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:22,484 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:22,484 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:22,484 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:22,484 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:22,485 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:22,485 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:22,485 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:22,485 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:22,485 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,485 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:22,485 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:22,486 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,486 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:22,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:22,487 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,487 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:22,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:22,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:22,487 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/be553bcc8b0d4de9b0955978b3273762 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/be553bcc8b0d4de9b0955978b3273762 2023-04-27 03:54:22,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:22,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:22,488 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:22,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10cbbaef[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:22,488 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,490 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,490 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:22,490 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:22,490 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:22,490 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,490 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:22,492 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d66a88af1ba39e44423282d0b046d5b3/B of d66a88af1ba39e44423282d0b046d5b3 into be553bcc8b0d4de9b0955978b3273762(size=12.2 K), total size for store is 21.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:22,492 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:22,492 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., storeName=d66a88af1ba39e44423282d0b046d5b3/B, priority=12, startTime=1682567662007; duration=0sec 2023-04-27 03:54:22,493 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:22,493 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 03:54:22,493 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,493 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:22,493 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:22,493 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:22,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:22,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:22,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567722483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:22,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567722483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:22,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52140 deadline: 1682567722488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,495 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58228 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 03:54:22,495 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): d66a88af1ba39e44423282d0b046d5b3/C is initiating minor compaction (all files) 2023-04-27 03:54:22,495 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d66a88af1ba39e44423282d0b046d5b3/C in TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:22,495 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/0f7229f465bf43a5ba2a1bcc3c887829, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/e82ded12040f4f668a30eaa63e137fe8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/91986859bcfc46e68ffccaaeec4c6b60, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/52961e2ca07c4ce9adcd71a9d01283e5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/251d34b133d34d6b923d7c40c0dbd153] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp, totalSize=56.9 K 2023-04-27 03:54:22,496 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 0f7229f465bf43a5ba2a1bcc3c887829, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1682567658946 2023-04-27 03:54:22,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:22,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:22,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,497 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting e82ded12040f4f668a30eaa63e137fe8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1682567659077 2023-04-27 03:54:22,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,497 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 91986859bcfc46e68ffccaaeec4c6b60, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682567659613 2023-04-27 03:54:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:22,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:22,498 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:22,498 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:22,498 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:22,498 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:22,498 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:22,498 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 52961e2ca07c4ce9adcd71a9d01283e5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1682567660242 2023-04-27 03:54:22,498 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 251d34b133d34d6b923d7c40c0dbd153, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1682567660347 2023-04-27 03:54:22,500 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,500 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:22,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,500 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:22,501 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:22,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:22,502 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:22,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,505 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:22,505 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:22,505 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:22,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:22,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:22,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:22,505 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,505 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,505 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:22,505 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,505 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,505 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:22,505 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:22,506 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:22,505 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,509 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): d66a88af1ba39e44423282d0b046d5b3#C#compaction#161 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:22,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:22,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567722584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:22,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567722595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:22,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567722596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,606 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:22,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:22,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:22,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:22,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:22,609 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:22,609 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:22,609 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:22,609 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:22,610 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,610 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,611 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:22,611 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:22,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:22,611 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,611 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:22,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:22,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:22,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:22,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:22,612 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:22,613 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:22,613 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:22,613 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:22,613 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:22,613 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:22,613 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:22,613 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:22,613 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:22,613 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:22,613 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:22,613 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,613 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:22,613 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:22,615 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,615 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:22,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:22,616 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,616 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:22,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:22,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:22,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:22,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:22,618 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:22,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31788ec5[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:22,618 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,619 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,619 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:22,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:22,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:22,620 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,620 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:22,621 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,621 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:22,621 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:22,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:22,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:22,621 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:22,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:22,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:22,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:22,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:22,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:22,623 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:22,624 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:22,624 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:22,624 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:22,624 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:22,625 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,625 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:22,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:22,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,625 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:22,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:22,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:22,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:22,626 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:22,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,628 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:22,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:22,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:22,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:22,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:22,631 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:22,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:22,631 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:22,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:22,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,632 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,632 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,632 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,632 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,632 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:22,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:22,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:22,732 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:22,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:22,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:22,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:22,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:22,735 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:22,735 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:22,735 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:22,735 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:22,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,737 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:22,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:22,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:22,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:22,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:22,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:22,738 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:22,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:22,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:22,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:22,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:22,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:22,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:22,738 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:22,738 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:22,739 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:22,739 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:22,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:22,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:22,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:22,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:22,740 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,740 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:22,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:22,740 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,741 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:22,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:22,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:22,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:22,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:22,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@705b00c5[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:22,742 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:22,742 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,743 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,743 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:22,743 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:22,743 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:22,743 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,743 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:22,746 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,746 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:22,746 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:22,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:22,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:22,746 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:22,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:22,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:22,749 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:22,749 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:22,749 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:22,749 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:22,749 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:22,750 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,750 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:22,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,750 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:22,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:22,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:22,751 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,752 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,758 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:22,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:22,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:22,758 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:22,759 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:22,759 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:22,759 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:22,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:22,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:22,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:22,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:22,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:22,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567722787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:22,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:22,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567722797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567722797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,837 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304272e68d86f82d442deaa0c7acf41ee3c2d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272e68d86f82d442deaa0c7acf41ee3c2d 2023-04-27 03:54:22,859 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:22,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:22,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:22,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:22,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:22,862 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:22,862 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:22,862 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:22,862 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:22,862 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,862 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:22,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:22,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:22,864 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,865 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,865 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:22,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:22,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:22,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:22,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:22,866 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:22,866 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:22,866 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:22,866 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:22,866 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:22,866 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:22,866 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:22,866 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:22,866 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:22,866 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:22,867 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:22,867 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,867 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:22,867 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:22,868 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:22,868 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,869 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:22,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:22,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66353b8e[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:22,870 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:22,870 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,871 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,871 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:22,871 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:22,871 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:22,871 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,871 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:22,873 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,873 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:22,873 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:22,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:22,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:22,873 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:22,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:22,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:22,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:22,875 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:22,875 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:22,875 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:22,875 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:22,875 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:22,878 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,878 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:22,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:22,878 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:22,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:22,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:22,878 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:22,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:22,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,880 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:22,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,882 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,882 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:22,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:22,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:22,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:22,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:22,882 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:22,882 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:22,882 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:22,882 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,883 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,883 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,883 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,883 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,883 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,882 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:22,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:22,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:22,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:22,924 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/71d5448509884ab588753bc75c4ae4d0 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/71d5448509884ab588753bc75c4ae4d0 2023-04-27 03:54:22,928 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d66a88af1ba39e44423282d0b046d5b3/C of d66a88af1ba39e44423282d0b046d5b3 into 71d5448509884ab588753bc75c4ae4d0(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:22,928 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:22,928 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., storeName=d66a88af1ba39e44423282d0b046d5b3/C, priority=11, startTime=1682567662008; duration=0sec 2023-04-27 03:54:22,929 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:22,983 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:22,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:22,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:22,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:22,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:22,986 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:22,986 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:22,986 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:22,986 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:22,987 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,987 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,988 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:22,988 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:22,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:22,988 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,988 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:22,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:22,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:22,989 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:22,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:22,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:22,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:22,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:22,990 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:22,990 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:22,990 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:22,990 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:22,990 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:22,990 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:22,990 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:22,990 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:22,990 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,990 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:22,990 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:22,992 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,992 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:22,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:22,992 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:22,992 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:22,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:22,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:22,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:22,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:22,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:22,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:22,993 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:22,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b64f965[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:22,993 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,000 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,001 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:23,001 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:23,001 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:23,001 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,001 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:23,002 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,002 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:23,002 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:23,003 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:23,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:23,005 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:23,005 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:23,005 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:23,005 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:23,005 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:23,007 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,007 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,007 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:23,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:23,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:23,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:23,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:23,007 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:23,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:23,008 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:23,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,010 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,010 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,010 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,010 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:23,010 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:23,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:23,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:23,010 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:23,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:23,010 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:23,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:23,011 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,011 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,011 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,011 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:23,011 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:23,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:23,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:23,091 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:23,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567723090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:23,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567723105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:23,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567723106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,111 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:23,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:23,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:23,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:23,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:23,115 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:23,115 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:23,115 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:23,115 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:23,115 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,115 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,117 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:23,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,118 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,118 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:23,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:23,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:23,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:23,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:23,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:23,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:23,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:23,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:23,121 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:23,121 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:23,121 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:23,121 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:23,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:23,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:23,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:23,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:23,123 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,123 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:23,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:23,123 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,123 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:23,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:23,125 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:23,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e9f446a[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:23,125 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,126 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,127 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:23,127 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:23,127 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:23,127 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,127 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:23,128 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,128 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:23,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:23,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:23,128 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:23,128 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:23,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:23,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:23,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:23,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:23,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:23,131 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:23,131 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:23,131 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:23,132 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:23,132 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:23,134 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,135 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:23,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:23,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,135 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:23,135 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:23,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:23,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:23,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:23,136 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:23,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,139 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,139 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:23,139 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,139 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:23,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:23,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:23,139 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,139 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,139 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:23,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:23,139 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:23,139 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:23,139 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:23,140 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,140 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:23,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:23,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:23,240 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:23,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:23,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:23,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:23,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:23,243 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=271, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/909907d368ac40faa7432ad75b201787 2023-04-27 03:54:23,243 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:23,243 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:23,243 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:23,243 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:23,244 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,244 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,245 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:23,245 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:23,245 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,245 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:23,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:23,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:23,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:23,249 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:23,249 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:23,249 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:23,249 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:23,249 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:23,249 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:23,249 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:23,250 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:23,250 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:23,250 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:23,250 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:23,250 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,250 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:23,250 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:23,260 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,260 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:23,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:23,260 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,260 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:23,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:23,261 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/1abece4aabde483b94257778e96dd68a 2023-04-27 03:54:23,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:23,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:23,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:23,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@766a3a54[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:23,263 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:23,263 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,264 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,264 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:23,264 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:23,264 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:23,269 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:23,269 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:23,270 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:23,270 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,270 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:23,270 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:23,270 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:23,270 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:23,270 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:23,270 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:23,270 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:23,272 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,272 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:23,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:23,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,272 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:23,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:23,273 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,274 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,277 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:23,277 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:23,277 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:23,277 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:23,277 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:23,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:23,277 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,277 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:23,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:23,277 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:23,277 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:23,277 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,277 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,278 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:23,319 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/ad74029298de45049031277a68acd07c as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/ad74029298de45049031277a68acd07c 2023-04-27 03:54:23,324 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d66a88af1ba39e44423282d0b046d5b3/A of d66a88af1ba39e44423282d0b046d5b3 into ad74029298de45049031277a68acd07c(size=25.8 K), total size for store is 44.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-27 03:54:23,324 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:23,324 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., storeName=d66a88af1ba39e44423282d0b046d5b3/A, priority=12, startTime=1682567662007; duration=1sec 2023-04-27 03:54:23,324 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:23,378 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:23,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:23,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:23,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:23,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:23,381 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:23,381 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:23,381 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:23,381 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:23,382 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,382 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,383 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:23,383 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:23,384 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,384 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:23,385 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:23,385 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:23,385 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:23,385 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:23,385 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:23,385 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:23,385 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:23,385 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:23,385 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:23,385 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:23,385 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:23,385 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:23,385 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:23,386 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,386 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:23,386 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:23,387 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:23,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:23,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:23,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39cc865c[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:23,389 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:23,389 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,390 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,391 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:23,391 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:23,391 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:23,391 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,391 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:23,393 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:23,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:23,393 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:23,393 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:23,394 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:23,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:23,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:23,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:23,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:23,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:23,396 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:23,396 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:23,396 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:23,396 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:23,396 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:23,398 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,398 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:23,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,398 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:23,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:23,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:23,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:23,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:23,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,399 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:23,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,400 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:23,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,403 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,403 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,403 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,403 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:23,403 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,403 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,403 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,404 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:23,404 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:23,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:23,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:23,404 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:23,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:23,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:23,404 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:23,404 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:23,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:23,504 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:23,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:23,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:23,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:23,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:23,508 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:23,508 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:23,508 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:23,508 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:23,509 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,509 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,510 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,510 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:23,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:23,510 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,510 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:23,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:23,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,511 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:23,511 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:23,511 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:23,511 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:23,511 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:23,511 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:23,512 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:23,512 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:23,512 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:23,512 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:23,512 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:23,512 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:23,512 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:23,512 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:23,512 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,512 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:23,512 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:23,514 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,514 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:23,515 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,516 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:23,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:23,517 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:23,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35efefd1[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:23,517 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,518 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,518 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:23,518 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:23,518 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,518 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:23,518 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:23,520 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,520 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:23,520 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:23,520 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:23,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:23,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:23,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:23,523 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:23,523 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:23,523 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:23,523 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:23,523 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:23,525 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:23,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:23,525 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,525 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:23,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:23,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,526 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:23,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,527 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:23,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,531 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,531 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,531 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,531 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,531 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:23,531 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:23,531 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:23,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:23,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:23,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:23,531 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:23,531 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,531 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,531 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:23,531 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:23,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:23,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:23,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52160 deadline: 1682567723592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:23,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52164 deadline: 1682567723608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:23,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52168 deadline: 1682567723611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,631 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:23,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:23,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:23,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:23,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:23,635 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:23,635 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:23,635 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:23,635 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:23,635 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,635 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,637 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:23,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:23,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:23,637 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,637 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,637 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:23,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:23,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:23,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:23,640 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:23,640 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:23,640 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:23,640 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:23,640 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:23,640 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:23,640 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:23,640 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:23,640 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:23,640 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:23,641 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:23,641 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,641 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:23,641 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:23,642 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,642 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:23,644 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,644 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:23,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:23,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11f74112[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:23,645 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:23,645 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,650 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,650 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:23,650 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,650 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:23,650 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:23,650 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:23,652 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:23,652 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,652 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:23,652 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:23,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:23,654 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:23,654 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:23,654 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:23,654 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:23,654 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:23,656 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:23,656 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,656 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:23,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:23,657 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:23,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:23,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,658 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:23,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:23,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,660 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,660 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,660 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,660 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:23,660 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:23,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:23,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:23,660 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:23,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:23,660 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,661 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,661 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:23,661 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:23,661 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:23,661 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:23,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:23,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:23,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:23,672 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/e057e8fdc33e4598aec10b77f5debd72 2023-04-27 03:54:23,677 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/909907d368ac40faa7432ad75b201787 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/909907d368ac40faa7432ad75b201787 2023-04-27 03:54:23,681 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/909907d368ac40faa7432ad75b201787, entries=350, sequenceid=271, filesize=52.9 K 2023-04-27 03:54:23,681 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/1abece4aabde483b94257778e96dd68a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/1abece4aabde483b94257778e96dd68a 2023-04-27 03:54:23,685 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/1abece4aabde483b94257778e96dd68a, entries=100, sequenceid=271, filesize=9.5 K 2023-04-27 03:54:23,686 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/e057e8fdc33e4598aec10b77f5debd72 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/e057e8fdc33e4598aec10b77f5debd72 2023-04-27 03:54:23,691 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/e057e8fdc33e4598aec10b77f5debd72, entries=100, sequenceid=271, filesize=9.5 K 2023-04-27 03:54:23,692 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for d66a88af1ba39e44423282d0b046d5b3 in 1277ms, sequenceid=271, compaction requested=true 2023-04-27 03:54:23,692 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:23,692 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:23,692 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:23,692 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:23,692 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:23,692 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:23,693 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 99582 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:23,693 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): d66a88af1ba39e44423282d0b046d5b3/A is initiating minor compaction (all files) 2023-04-27 03:54:23,693 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 31998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:23,693 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d66a88af1ba39e44423282d0b046d5b3/A in TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:23,693 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): d66a88af1ba39e44423282d0b046d5b3/B is initiating minor compaction (all files) 2023-04-27 03:54:23,693 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/ad74029298de45049031277a68acd07c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/5f0a6951ac814b769bd1ec7f172f7dbd, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/909907d368ac40faa7432ad75b201787] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp, totalSize=97.2 K 2023-04-27 03:54:23,694 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d66a88af1ba39e44423282d0b046d5b3/B in TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:23,694 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/be553bcc8b0d4de9b0955978b3273762, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/4479c57b5ff542ca904782887d659452, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/1abece4aabde483b94257778e96dd68a] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp, totalSize=31.2 K 2023-04-27 03:54:23,694 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting ad74029298de45049031277a68acd07c, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1682567660242 2023-04-27 03:54:23,695 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting be553bcc8b0d4de9b0955978b3273762, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1682567660242 2023-04-27 03:54:23,695 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 5f0a6951ac814b769bd1ec7f172f7dbd, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1682567660347 2023-04-27 03:54:23,695 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 4479c57b5ff542ca904782887d659452, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1682567660347 2023-04-27 03:54:23,695 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 909907d368ac40faa7432ad75b201787, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682567662395 2023-04-27 03:54:23,696 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 1abece4aabde483b94257778e96dd68a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682567662410 2023-04-27 03:54:23,725 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): d66a88af1ba39e44423282d0b046d5b3#B#compaction#165 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:23,741 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): d66a88af1ba39e44423282d0b046d5b3#A#compaction#164 average throughput is 0.53 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:23,758 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/56f953307da24777861c852e8c4459e2 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/56f953307da24777861c852e8c4459e2 2023-04-27 03:54:23,761 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:23,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:23,764 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d66a88af1ba39e44423282d0b046d5b3/B of d66a88af1ba39e44423282d0b046d5b3 into 56f953307da24777861c852e8c4459e2(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:23,764 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:23,764 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., storeName=d66a88af1ba39e44423282d0b046d5b3/B, priority=13, startTime=1682567663692; duration=0sec 2023-04-27 03:54:23,764 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:23,764 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-27 03:54:23,765 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-27 03:54:23,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:23,765 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-27 03:54:23,765 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. because compaction request was cancelled 2023-04-27 03:54:23,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:23,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:23,766 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:23,766 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:23,766 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:23,766 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:23,766 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,766 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,769 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,769 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:23,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:23,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:23,769 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:23,769 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:23,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:23,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:23,773 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:23,773 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:23,773 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:23,773 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:23,773 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:23,773 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:23,773 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:23,774 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:23,774 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:23,774 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d66a88af1ba39e44423282d0b046d5b3 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-27 03:54:23,774 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=A 2023-04-27 03:54:23,774 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:23,774 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=B 2023-04-27 03:54:23,774 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:23,774 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=C 2023-04-27 03:54:23,774 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:23,802 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304279615bb7d05c346ed8207100d63ad4165 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304279615bb7d05c346ed8207100d63ad4165 2023-04-27 03:54:24,210 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=311, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/dfd120972de44a9eaca1cbbceb7f3654 2023-04-27 03:54:24,228 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/850210a3fee34082ba33a108919fc096 2023-04-27 03:54:24,240 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/ac17cc391be240559c55732aa70ab185 2023-04-27 03:54:24,244 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/dfd120972de44a9eaca1cbbceb7f3654 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/dfd120972de44a9eaca1cbbceb7f3654 2023-04-27 03:54:24,248 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/dfd120972de44a9eaca1cbbceb7f3654, entries=150, sequenceid=311, filesize=25.4 K 2023-04-27 03:54:24,249 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/850210a3fee34082ba33a108919fc096 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/850210a3fee34082ba33a108919fc096 2023-04-27 03:54:24,253 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/850210a3fee34082ba33a108919fc096, entries=150, sequenceid=311, filesize=11.9 K 2023-04-27 03:54:24,254 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/ac17cc391be240559c55732aa70ab185 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/ac17cc391be240559c55732aa70ab185 2023-04-27 03:54:24,260 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/ac17cc391be240559c55732aa70ab185, entries=150, sequenceid=311, filesize=11.9 K 2023-04-27 03:54:24,261 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=0 B/0 for d66a88af1ba39e44423282d0b046d5b3 in 487ms, sequenceid=311, compaction requested=true 2023-04-27 03:54:24,261 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:24,261 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:24,261 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:24,261 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:24,261 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,261 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:24,261 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:24,263 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:24,263 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:24,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:24,263 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:24,263 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:24,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:24,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:24,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:24,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:24,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:24,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36e871e2[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:24,265 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:24,265 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:24,266 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:24,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:24,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:24,267 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:24,267 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:24,267 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:24,267 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,267 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:24,268 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:24,268 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,268 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:24,268 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:24,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:24,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:24,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:24,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:24,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:24,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:24,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:24,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:24,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:24,271 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:24,271 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:24,271 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:24,271 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:24,271 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:24,274 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:24,274 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:24,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:24,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:24,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:24,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:24,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:24,274 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:24,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:24,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:24,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:24,275 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:24,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:24,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:24,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:24,276 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:24,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:24,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,278 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,278 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:24,278 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:24,278 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:24,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:24,278 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:24,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:24,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:24,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:24,278 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:24,279 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:24,279 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:24,279 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:24,279 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:24,279 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:24,278 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:24,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:24,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:24,379 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:24,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:24,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:24,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:24,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:24,383 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:24,383 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:24,383 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:24,383 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:24,384 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:24,384 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:24,385 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:24,385 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:24,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:24,385 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,386 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:24,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:24,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:24,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:24,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:24,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:24,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:24,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:24,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:24,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:24,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:24,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:24,387 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:24,387 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:24,387 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:24,387 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:24,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:24,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:24,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:24,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:24,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,389 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:24,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:24,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:24,389 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:24,389 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:24,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:24,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:24,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:24,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:24,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:24,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bcbef05[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:24,391 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:24,391 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:24,394 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:24,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:24,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:24,394 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:24,394 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:24,394 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:24,394 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,394 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:24,396 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,396 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:24,396 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:24,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:24,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:24,396 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:24,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:24,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:24,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:24,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:24,399 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:24,399 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:24,399 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:24,399 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:24,399 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:24,401 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:24,401 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:24,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:24,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:24,401 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:24,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:24,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:24,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:24,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:24,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:24,402 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:24,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:24,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:24,404 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:24,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:24,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,406 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,406 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:24,406 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:24,406 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:24,406 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:24,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:24,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:24,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:24,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:24,406 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,406 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:24,407 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:24,407 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:24,407 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:24,407 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:24,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:24,407 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:24,507 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:24,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:24,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:24,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:24,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:24,511 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:24,511 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:24,511 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:24,511 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:24,512 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:24,512 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:24,514 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:24,514 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:24,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:24,514 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,514 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:24,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:24,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:24,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:24,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:24,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:24,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:24,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:24,520 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:24,520 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:24,520 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:24,524 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:24,524 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:24,524 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:24,524 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:24,545 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d66a88af1ba39e44423282d0b046d5b3 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-27 03:54:24,546 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=A 2023-04-27 03:54:24,546 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:24,546 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=B 2023-04-27 03:54:24,546 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:24,546 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=C 2023-04-27 03:54:24,546 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:24,577 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/c6553da57498463eb5afaa4a67661b0d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/c6553da57498463eb5afaa4a67661b0d 2023-04-27 03:54:24,581 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427e3807a7cad5f4464bad06deb128e9ea1 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427e3807a7cad5f4464bad06deb128e9ea1 2023-04-27 03:54:24,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:24,584 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:24,585 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d66a88af1ba39e44423282d0b046d5b3/A of d66a88af1ba39e44423282d0b046d5b3 into c6553da57498463eb5afaa4a67661b0d(size=26.0 K), total size for store is 51.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:24,585 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:24,585 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., storeName=d66a88af1ba39e44423282d0b046d5b3/A, priority=13, startTime=1682567663692; duration=0sec 2023-04-27 03:54:24,585 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:24,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1682567664588, max timestamp=1682567664610Num uniques 859; 2023-04-27 03:54:24,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-27 03:54:24,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1682567664588, max timestamp=1682567664610Num uniques 859; 2023-04-27 03:54:24,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1682567664588, max timestamp=1682567664610Num uniques 859; 2023-04-27 03:54:24,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-27 03:54:24,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-27 03:54:24,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682567664628, max timestamp=1682567664637Num uniques 1118; 2023-04-27 03:54:24,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-27 03:54:24,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682567664628, max timestamp=1682567664637Num uniques 1118; 2023-04-27 03:54:24,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-27 03:54:24,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682567664628, max timestamp=1682567664637Num uniques 1118; 2023-04-27 03:54:24,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-27 03:54:24,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682567664659, max timestamp=1682567664663Num uniques 1127; 2023-04-27 03:54:24,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-27 03:54:24,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682567664659, max timestamp=1682567664663Num uniques 1127; 2023-04-27 03:54:24,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-27 03:54:24,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682567664659, max timestamp=1682567664663Num uniques 1127; 2023-04-27 03:54:24,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-27 03:54:24,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682567664677, max timestamp=1682567664690Num uniques 1086; 2023-04-27 03:54:24,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-27 03:54:24,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682567664686, max timestamp=1682567664690Num uniques 1086; 2023-04-27 03:54:24,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-27 03:54:24,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682567664686, max timestamp=1682567664690Num uniques 1086; 2023-04-27 03:54:24,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-27 03:54:24,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1682567664705, max timestamp=1682567664719Num uniques 1145; 2023-04-27 03:54:24,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-27 03:54:24,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1682567664705, max timestamp=1682567664719Num uniques 1045; 2023-04-27 03:54:24,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-27 03:54:24,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1682567664705, max timestamp=1682567664719Num uniques 1045; 2023-04-27 03:54:24,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-27 03:54:24,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13920, totalHeapSize=31856, min timestamp=1682567664733, max timestamp=1682567664746Num uniques 1154; 2023-04-27 03:54:24,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13920, totalHeapSize=31856, min timestamp=1682567664733, max timestamp=1682567664746Num uniques 1154; 2023-04-27 03:54:24,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=13920, suffix heap size=147984,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=304 2023-04-27 03:54:24,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=13920, suffix heap size=147984,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=304 2023-04-27 03:54:24,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13920, totalHeapSize=31856, min timestamp=1682567664733, max timestamp=1682567664746Num uniques 1154; 2023-04-27 03:54:24,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=13920, suffix heap size=147984,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=304 2023-04-27 03:54:24,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=363, cellSize=16618, totalHeapSize=37992, min timestamp=1682567664758, max timestamp=1682567664774Num uniques 1163; 2023-04-27 03:54:24,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=363, cellSize=16618, totalHeapSize=37992, min timestamp=1682567664758, max timestamp=1682567664774Num uniques 1163; 2023-04-27 03:54:24,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=53258, new segment data size=16618, suffix heap size=148920,new segment heap size=37992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=363 2023-04-27 03:54:24,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53258, new segment data size=16618, suffix heap size=148920,new segment heap size=37992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=363 2023-04-27 03:54:24,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=363, cellSize=16618, totalHeapSize=37992, min timestamp=1682567664758, max timestamp=1682567664774Num uniques 1163; 2023-04-27 03:54:24,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53258, new segment data size=16618, suffix heap size=148920,new segment heap size=37992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=363 2023-04-27 03:54:24,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682567664789, max timestamp=1682567664798Num uniques 1222; 2023-04-27 03:54:24,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=55962, new segment data size=10162, suffix heap size=155056,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1222, new segment cells count=222 2023-04-27 03:54:24,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682567664789, max timestamp=1682567664798Num uniques 1222; 2023-04-27 03:54:24,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=55962, new segment data size=10162, suffix heap size=155056,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1222, new segment cells count=222 2023-04-27 03:54:24,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682567664789, max timestamp=1682567664798Num uniques 1222; 2023-04-27 03:54:24,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=55962, new segment data size=10162, suffix heap size=155056,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1222, new segment cells count=222 2023-04-27 03:54:24,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682567664797, max timestamp=1682567664823Num uniques 1081; 2023-04-27 03:54:24,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-27 03:54:24,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682567664797, max timestamp=1682567664823Num uniques 1081; 2023-04-27 03:54:24,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-27 03:54:24,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682567664797, max timestamp=1682567664823Num uniques 1081; 2023-04-27 03:54:24,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-27 03:54:24,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=490, cellSize=22440, totalHeapSize=51200, min timestamp=1682567664835, max timestamp=1682567664847Num uniques 1090; 2023-04-27 03:54:24,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=22440, suffix heap size=141328,new segment heap size=51200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=490 2023-04-27 03:54:24,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=490, cellSize=22440, totalHeapSize=51200, min timestamp=1682567664835, max timestamp=1682567664847Num uniques 1090; 2023-04-27 03:54:24,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=22440, suffix heap size=141328,new segment heap size=51200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=490 2023-04-27 03:54:24,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=490, cellSize=22440, totalHeapSize=51200, min timestamp=1682567664835, max timestamp=1682567664847Num uniques 1090; 2023-04-27 03:54:24,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=22440, suffix heap size=141328,new segment heap size=51200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=490 2023-04-27 03:54:24,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:24,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1682567724865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:24,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1682567724866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:24,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52164 deadline: 1682567724866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:24,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52168 deadline: 1682567724868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:24,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52150 deadline: 1682567724869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,975 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:24,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1682567724974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:24,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1682567724974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:24,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52164 deadline: 1682567724975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:24,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52168 deadline: 1682567724978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:24,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:24,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52150 deadline: 1682567724979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,007 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=317, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/951dd297ddd0440ab2d6d3dea0ee6b9a 2023-04-27 03:54:25,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:25,180 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:25,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52164 deadline: 1682567725177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1682567725178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:25,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1682567725178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,187 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:25,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52168 deadline: 1682567725184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:25,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52150 deadline: 1682567725185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,422 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/62d575e93c6f4b76b8c9ff1e8d1a2d53 2023-04-27 03:54:25,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:25,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52164 deadline: 1682567725481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:25,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1682567725483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:25,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1682567725484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:25,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52168 deadline: 1682567725491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:25,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52150 deadline: 1682567725492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,850 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/4503675ee1974f838f7ac0418a203e33 2023-04-27 03:54:25,856 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/951dd297ddd0440ab2d6d3dea0ee6b9a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/951dd297ddd0440ab2d6d3dea0ee6b9a 2023-04-27 03:54:25,860 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/951dd297ddd0440ab2d6d3dea0ee6b9a, entries=100, sequenceid=317, filesize=18.6 K 2023-04-27 03:54:25,861 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/62d575e93c6f4b76b8c9ff1e8d1a2d53 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/62d575e93c6f4b76b8c9ff1e8d1a2d53 2023-04-27 03:54:25,865 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/62d575e93c6f4b76b8c9ff1e8d1a2d53, entries=100, sequenceid=317, filesize=9.5 K 2023-04-27 03:54:25,866 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/4503675ee1974f838f7ac0418a203e33 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/4503675ee1974f838f7ac0418a203e33 2023-04-27 03:54:25,870 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/4503675ee1974f838f7ac0418a203e33, entries=100, sequenceid=317, filesize=9.5 K 2023-04-27 03:54:25,871 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=174.43 KB/178620 for d66a88af1ba39e44423282d0b046d5b3 in 1347ms, sequenceid=317, compaction requested=true 2023-04-27 03:54:25,871 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:25,871 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:25,871 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:25,871 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:25,871 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,871 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:25,871 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:25,873 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:25,873 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:25,873 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:25,873 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:25,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:25,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@211004ca[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:25,875 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:25,875 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:25,876 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:25,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:25,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:25,876 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:25,876 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:25,876 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:25,877 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,877 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:25,882 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:25,883 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:25,882 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,883 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:25,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:25,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:25,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:25,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:25,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:25,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:25,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:25,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:25,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:25,885 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:25,885 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:25,885 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:25,885 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:25,885 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:25,887 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:25,887 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:25,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:25,887 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:25,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:25,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:25,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:25,888 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:25,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:25,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:25,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:25,889 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:25,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:25,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,893 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:25,893 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,893 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:25,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:25,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:25,893 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:25,893 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:25,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:25,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:25,893 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:25,893 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:25,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:25,893 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:25,893 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:25,893 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:25,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:25,893 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:25,893 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:25,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:25,988 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d66a88af1ba39e44423282d0b046d5b3 3/3 column families, dataSize=181.14 KB heapSize=494.11 KB 2023-04-27 03:54:25,988 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=A 2023-04-27 03:54:25,988 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-27 03:54:25,988 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=B 2023-04-27 03:54:25,988 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-27 03:54:25,988 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=C 2023-04-27 03:54:25,988 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-27 03:54:25,994 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:25,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:25,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:25,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:25,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:25,999 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:25,999 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:25,999 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:25,999 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:25,999 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:25,999 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,001 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:26,001 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:26,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:26,001 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,001 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:26,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:26,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:26,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:26,005 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:26,005 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:26,005 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:26,005 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:26,005 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:26,005 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:26,005 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:26,005 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:26,005 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:26,005 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:26,005 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:26,005 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,006 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:26,006 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:26,007 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,008 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:26,008 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,008 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:26,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:26,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:26,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ac43e2b[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:26,009 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:26,009 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,010 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,011 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:26,011 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:26,011 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:26,011 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,011 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:26,012 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,012 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:26,012 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:26,012 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:26,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:26,014 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:26,014 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:26,014 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:26,014 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:26,014 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:26,015 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,015 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:26,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,016 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,016 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:26,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,017 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,020 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,020 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:26,020 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:26,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:26,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:26,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:26,020 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:26,020 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:26,020 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:26,021 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:26,021 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:26,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:26,021 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,021 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,021 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,021 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:26,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52168 deadline: 1682567725998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:26,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52150 deadline: 1682567726000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:26,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1682567726020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:26,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1682567726020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:26,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52164 deadline: 1682567726020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,121 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:26,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:26,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:26,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:26,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:26,124 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:26,124 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:26,124 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:26,124 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:26,124 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,124 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,126 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,126 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:26,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:26,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:26,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,126 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,126 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:26,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:26,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,127 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:26,127 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:26,127 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:26,127 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:26,127 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:26,127 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:26,128 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:26,128 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:26,128 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:26,128 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:26,128 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:26,128 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:26,128 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:26,128 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:26,128 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,128 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:26,128 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:26,130 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,130 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:26,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:26,130 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,130 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:26,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:26,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:26,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:26,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:26,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:26,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:26,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1682567726126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52164 deadline: 1682567726127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:26,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fd629f0[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:26,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1682567726126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,132 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:26,136 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,137 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:26,137 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:26,137 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:26,138 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,138 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:26,140 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,140 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:26,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:26,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:26,140 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:26,140 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:26,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:26,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:26,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:26,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:26,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:26,143 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:26,143 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:26,143 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:26,143 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:26,143 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:26,146 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,146 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:26,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,146 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:26,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:26,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,147 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:26,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,148 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,150 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,150 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,150 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,150 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:26,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:26,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:26,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:26,150 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:26,150 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,150 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:26,151 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,151 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,151 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:26,151 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:26,151 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:26,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:26,251 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:26,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:26,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:26,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:26,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:26,254 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:26,254 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:26,254 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:26,254 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:26,255 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,255 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,256 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:26,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:26,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:26,256 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,257 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,257 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:26,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,257 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:26,257 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:26,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:26,258 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:26,258 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:26,258 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:26,258 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:26,258 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:26,258 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:26,259 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:26,259 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:26,259 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:26,259 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:26,259 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:26,259 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,259 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:26,259 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:26,262 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,262 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:26,262 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,262 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:26,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:26,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:26,263 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:26,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bd142c8[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:26,263 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,265 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,265 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:26,265 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:26,265 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,265 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:26,265 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:26,267 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,267 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:26,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:26,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:26,267 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:26,267 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:26,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:26,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:26,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:26,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:26,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:26,269 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:26,269 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:26,269 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:26,270 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:26,270 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:26,271 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,271 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:26,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:26,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:26,272 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,271 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:26,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:26,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:26,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,273 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:26,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,278 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,278 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,278 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,278 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,278 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,279 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:26,278 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:26,279 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:26,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:26,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:26,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:26,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:26,279 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,279 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:26,279 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:26,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:26,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:26,279 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:26,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:26,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1682567726334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:26,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52164 deadline: 1682567726334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:26,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1682567726337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,379 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:26,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:26,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:26,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:26,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:26,383 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:26,383 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:26,383 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:26,383 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:26,383 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,383 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,385 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,385 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:26,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:26,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:26,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,385 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,385 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:26,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:26,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:26,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:26,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:26,386 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:26,386 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:26,386 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:26,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:26,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:26,387 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:26,387 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:26,387 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:26,387 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:26,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:26,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:26,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:26,387 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:26,388 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,388 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:26,389 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,389 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:26,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:26,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:26,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:26,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:26,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ac6292c[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:26,394 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:26,394 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,395 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,395 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:26,395 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:26,396 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:26,396 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,396 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:26,398 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:26,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:26,398 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:26,398 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:26,398 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:26,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:26,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:26,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:26,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:26,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:26,400 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:26,400 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:26,400 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:26,400 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:26,401 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:26,402 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:26,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:26,403 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,403 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:26,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:26,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,404 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:26,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,405 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,408 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:26,408 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:26,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:26,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:26,408 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:26,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:26,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:26,408 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,408 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,408 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,408 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,408 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,408 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,408 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,408 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:26,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:26,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:26,408 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:26,410 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042738c6ee75434142eaa18a3ed08875f2d2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042738c6ee75434142eaa18a3ed08875f2d2 2023-04-27 03:54:26,508 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:26,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:26,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:26,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:26,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:26,512 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:26,512 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:26,512 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:26,512 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:26,513 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,513 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,514 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:26,514 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:26,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:26,514 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,514 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:26,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:26,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:26,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:26,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:26,516 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:26,516 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:26,516 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:26,516 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:26,516 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:26,516 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:26,516 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:26,516 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:26,516 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:26,516 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:26,516 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:26,516 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,516 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:26,516 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:26,524 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,524 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:26,525 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,525 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:26,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:26,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:26,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:26,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:26,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41587739[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:26,526 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:26,526 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,528 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,528 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:26,528 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:26,528 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:26,528 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,528 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:26,530 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:26,530 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,530 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:26,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:26,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:26,530 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:26,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:26,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:26,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:26,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:26,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:26,533 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:26,533 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:26,533 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:26,533 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:26,533 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:26,535 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,535 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,535 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:26,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:26,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:26,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:26,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:26,535 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:26,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:26,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,536 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:26,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,541 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:26,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:26,541 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,541 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,541 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,541 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,541 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,541 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,541 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:26,541 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:26,541 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:26,541 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:26,542 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:26,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:26,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:26,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:26,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:26,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:26,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:26,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1682567726637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:26,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52164 deadline: 1682567726639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,642 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:26,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:26,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:26,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1682567726640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:26,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:26,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:26,645 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:26,645 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:26,645 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:26,645 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:26,646 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,646 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,647 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:26,647 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:26,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:26,647 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,647 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:26,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:26,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:26,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:26,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:26,649 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:26,649 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:26,649 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:26,650 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:26,650 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:26,650 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:26,650 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:26,650 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:26,650 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:26,650 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:26,650 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:26,650 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,650 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:26,650 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:26,652 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,652 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:26,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:26,652 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,652 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:26,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:26,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:26,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f8f5f4b[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:26,654 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:26,654 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,656 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,656 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,656 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:26,656 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:26,656 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:26,656 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:26,659 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,659 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:26,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:26,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:26,659 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:26,659 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:26,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:26,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:26,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:26,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:26,662 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:26,662 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:26,662 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:26,662 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:26,662 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:26,664 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,664 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:26,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:26,664 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:26,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:26,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:26,665 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:26,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:26,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,667 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,672 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,672 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:26,672 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,672 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:26,672 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,672 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,672 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:26,672 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:26,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:26,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:26,672 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:26,672 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,672 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:26,672 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:26,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:26,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:26,772 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:26,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:26,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:26,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:26,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:26,776 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:26,776 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:26,776 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:26,776 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:26,776 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,776 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,779 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:26,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:26,779 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:26,779 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,779 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:26,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:26,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:26,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:26,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:26,781 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:26,781 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:26,781 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:26,781 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:26,781 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:26,781 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:26,781 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:26,781 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:26,781 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:26,781 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:26,781 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:26,781 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,781 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:26,781 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:26,783 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,783 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:26,783 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,783 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:26,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:26,785 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:26,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e898b70[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:26,785 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,786 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,786 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:26,786 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:26,786 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:26,786 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,786 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:26,787 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:26,787 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,787 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:26,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:26,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:26,787 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:26,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:26,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:26,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:26,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:26,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:26,790 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:26,790 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:26,790 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:26,790 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:26,790 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:26,792 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,792 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:26,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:26,792 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:26,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:26,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:26,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:26,793 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,794 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:26,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:26,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,797 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,797 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,797 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,797 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,797 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,797 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:26,797 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:26,797 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:26,797 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:26,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:26,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:26,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:26,797 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:26,797 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:26,797 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:26,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:26,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:26,818 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=510, memsize=60.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/e6cf51cc68684827bdbf5eb1b1e4ef54 2023-04-27 03:54:26,840 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/e31b8bc0f16f4c10b498eb798f26f37a 2023-04-27 03:54:26,877 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/5e7f958a13454b1e810f6657f4b7419a 2023-04-27 03:54:26,888 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/e6cf51cc68684827bdbf5eb1b1e4ef54 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/e6cf51cc68684827bdbf5eb1b1e4ef54 2023-04-27 03:54:26,895 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/e6cf51cc68684827bdbf5eb1b1e4ef54, entries=200, sequenceid=510, filesize=32.3 K 2023-04-27 03:54:26,896 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/e31b8bc0f16f4c10b498eb798f26f37a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/e31b8bc0f16f4c10b498eb798f26f37a 2023-04-27 03:54:26,897 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:26,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:26,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:26,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:26,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:26,903 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:26,903 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:26,903 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:26,903 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:26,904 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,904 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,905 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/e31b8bc0f16f4c10b498eb798f26f37a, entries=150, sequenceid=510, filesize=11.9 K 2023-04-27 03:54:26,905 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:26,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:26,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:26,905 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,906 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:26,906 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:26,906 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/5e7f958a13454b1e810f6657f4b7419a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/5e7f958a13454b1e810f6657f4b7419a 2023-04-27 03:54:26,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:26,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:26,912 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:26,912 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:26,914 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/5e7f958a13454b1e810f6657f4b7419a, entries=150, sequenceid=510, filesize=11.9 K 2023-04-27 03:54:26,914 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:26,914 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:26,914 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~181.14 KB/185490, heapSize ~494.06 KB/505920, currentSize=20.13 KB/20610 for d66a88af1ba39e44423282d0b046d5b3 in 926ms, sequenceid=510, compaction requested=true 2023-04-27 03:54:26,915 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:26,915 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:26,915 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:26,915 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 03:54:26,915 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:26,915 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 03:54:26,917 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104706 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 03:54:26,917 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46830 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 03:54:26,917 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): d66a88af1ba39e44423282d0b046d5b3/A is initiating minor compaction (all files) 2023-04-27 03:54:26,917 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): d66a88af1ba39e44423282d0b046d5b3/B is initiating minor compaction (all files) 2023-04-27 03:54:26,917 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d66a88af1ba39e44423282d0b046d5b3/A in TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:26,917 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d66a88af1ba39e44423282d0b046d5b3/B in TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:26,917 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/c6553da57498463eb5afaa4a67661b0d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/dfd120972de44a9eaca1cbbceb7f3654, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/951dd297ddd0440ab2d6d3dea0ee6b9a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/e6cf51cc68684827bdbf5eb1b1e4ef54] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp, totalSize=102.3 K 2023-04-27 03:54:26,917 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/56f953307da24777861c852e8c4459e2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/850210a3fee34082ba33a108919fc096, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/62d575e93c6f4b76b8c9ff1e8d1a2d53, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/e31b8bc0f16f4c10b498eb798f26f37a] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp, totalSize=45.7 K 2023-04-27 03:54:26,918 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting c6553da57498463eb5afaa4a67661b0d, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682567660249 2023-04-27 03:54:26,918 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 56f953307da24777861c852e8c4459e2, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682567660249 2023-04-27 03:54:26,918 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting dfd120972de44a9eaca1cbbceb7f3654, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1682567662472 2023-04-27 03:54:26,918 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 850210a3fee34082ba33a108919fc096, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1682567662472 2023-04-27 03:54:26,918 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 951dd297ddd0440ab2d6d3dea0ee6b9a, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682567664514 2023-04-27 03:54:26,919 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 62d575e93c6f4b76b8c9ff1e8d1a2d53, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682567664514 2023-04-27 03:54:26,919 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting e6cf51cc68684827bdbf5eb1b1e4ef54, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1682567664864 2023-04-27 03:54:26,919 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting e31b8bc0f16f4c10b498eb798f26f37a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1682567664866 2023-04-27 03:54:26,924 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d66a88af1ba39e44423282d0b046d5b3 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-27 03:54:26,924 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=A 2023-04-27 03:54:26,924 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:26,924 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=B 2023-04-27 03:54:26,924 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:26,925 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=C 2023-04-27 03:54:26,925 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:26,941 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): d66a88af1ba39e44423282d0b046d5b3#B#compaction#175 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:26,970 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): d66a88af1ba39e44423282d0b046d5b3#A#compaction#176 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:26,976 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/83116b4a1eaa4231bab25fef8f75081f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/83116b4a1eaa4231bab25fef8f75081f 2023-04-27 03:54:26,981 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427823685dabc924758bc1096956e298c9a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427823685dabc924758bc1096956e298c9a 2023-04-27 03:54:26,984 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d66a88af1ba39e44423282d0b046d5b3/B of d66a88af1ba39e44423282d0b046d5b3 into 83116b4a1eaa4231bab25fef8f75081f(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:26,984 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:26,984 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., storeName=d66a88af1ba39e44423282d0b046d5b3/B, priority=12, startTime=1682567666915; duration=0sec 2023-04-27 03:54:26,984 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:26,984 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 03:54:26,986 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 56504 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 03:54:26,987 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): d66a88af1ba39e44423282d0b046d5b3/C is initiating minor compaction (all files) 2023-04-27 03:54:26,987 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d66a88af1ba39e44423282d0b046d5b3/C in TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:26,987 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/71d5448509884ab588753bc75c4ae4d0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/e057e8fdc33e4598aec10b77f5debd72, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/ac17cc391be240559c55732aa70ab185, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/4503675ee1974f838f7ac0418a203e33, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/5e7f958a13454b1e810f6657f4b7419a] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp, totalSize=55.2 K 2023-04-27 03:54:26,987 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 71d5448509884ab588753bc75c4ae4d0, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1682567660249 2023-04-27 03:54:26,988 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting e057e8fdc33e4598aec10b77f5debd72, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1682567662410 2023-04-27 03:54:26,988 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting ac17cc391be240559c55732aa70ab185, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1682567662472 2023-04-27 03:54:26,989 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 4503675ee1974f838f7ac0418a203e33, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682567664514 2023-04-27 03:54:26,989 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 5e7f958a13454b1e810f6657f4b7419a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1682567664866 2023-04-27 03:54:26,998 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): d66a88af1ba39e44423282d0b046d5b3#C#compaction#178 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:27,015 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/122250d39b384ab29709f768e0f1dfa0 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/122250d39b384ab29709f768e0f1dfa0 2023-04-27 03:54:27,019 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d66a88af1ba39e44423282d0b046d5b3/C of d66a88af1ba39e44423282d0b046d5b3 into 122250d39b384ab29709f768e0f1dfa0(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:27,019 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:27,019 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., storeName=d66a88af1ba39e44423282d0b046d5b3/C, priority=11, startTime=1682567666915; duration=0sec 2023-04-27 03:54:27,020 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:27,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:27,034 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:27,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567667049, max timestamp=1682567667063Num uniques 859; 2023-04-27 03:54:27,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567667049, max timestamp=1682567667063Num uniques 859; 2023-04-27 03:54:27,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:54:27,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567667049, max timestamp=1682567667063Num uniques 859; 2023-04-27 03:54:27,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:54:27,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:54:27,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1682567667083, max timestamp=1682567667095Num uniques 1018; 2023-04-27 03:54:27,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-27 03:54:27,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1682567667083, max timestamp=1682567667095Num uniques 1018; 2023-04-27 03:54:27,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-27 03:54:27,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682567667086, max timestamp=1682567667095Num uniques 1018; 2023-04-27 03:54:27,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-27 03:54:27,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:27,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682567667107, max timestamp=1682567667140Num uniques 1177; 2023-04-27 03:54:27,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52150 deadline: 1682567727141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:27,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=8102, suffix heap size=150376,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=177 2023-04-27 03:54:27,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682567667107, max timestamp=1682567667140Num uniques 1027; 2023-04-27 03:54:27,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-27 03:54:27,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682567667107, max timestamp=1682567667140Num uniques 1177; 2023-04-27 03:54:27,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=8102, suffix heap size=150376,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=177 2023-04-27 03:54:27,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=436, cellSize=19966, totalHeapSize=45584, min timestamp=1682567667151, max timestamp=1682567667160Num uniques 1036; 2023-04-27 03:54:27,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=19966, suffix heap size=135712,new segment heap size=45584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=436 2023-04-27 03:54:27,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=436, cellSize=19966, totalHeapSize=45584, min timestamp=1682567667151, max timestamp=1682567667160Num uniques 1036; 2023-04-27 03:54:27,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=19966, suffix heap size=135712,new segment heap size=45584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=436 2023-04-27 03:54:27,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=436, cellSize=19966, totalHeapSize=45584, min timestamp=1682567667151, max timestamp=1682567667160Num uniques 1036; 2023-04-27 03:54:27,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=19966, suffix heap size=135712,new segment heap size=45584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=436 2023-04-27 03:54:27,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:27,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1682567667173, max timestamp=1682567667180Num uniques 1295; 2023-04-27 03:54:27,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=59310, new segment data size=13510, suffix heap size=162648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1295, new segment cells count=295 2023-04-27 03:54:27,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1682567667173, max timestamp=1682567667180Num uniques 1295; 2023-04-27 03:54:27,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=59310, new segment data size=13510, suffix heap size=162648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1295, new segment cells count=295 2023-04-27 03:54:27,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1682567667173, max timestamp=1682567667180Num uniques 1295; 2023-04-27 03:54:27,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1682567727177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:27,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=59310, new segment data size=13510, suffix heap size=162648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1295, new segment cells count=295 2023-04-27 03:54:27,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1682567667189, max timestamp=1682567667209Num uniques 1154; 2023-04-27 03:54:27,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1682567667189, max timestamp=1682567667209Num uniques 1154; 2023-04-27 03:54:27,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-27 03:54:27,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-27 03:54:27,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1682567667189, max timestamp=1682567667209Num uniques 1154; 2023-04-27 03:54:27,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-27 03:54:27,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682567667220, max timestamp=1682567667231Num uniques 1113; 2023-04-27 03:54:27,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-27 03:54:27,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682567667220, max timestamp=1682567667231Num uniques 1113; 2023-04-27 03:54:27,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682567667220, max timestamp=1682567667231Num uniques 1113; 2023-04-27 03:54:27,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-27 03:54:27,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-27 03:54:27,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682567667249, max timestamp=1682567667257Num uniques 1072; 2023-04-27 03:54:27,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-27 03:54:27,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682567667249, max timestamp=1682567667257Num uniques 1072; 2023-04-27 03:54:27,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-27 03:54:27,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682567667249, max timestamp=1682567667257Num uniques 1072; 2023-04-27 03:54:27,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-27 03:54:27,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1682567667269, max timestamp=1682567667276Num uniques 1031; 2023-04-27 03:54:27,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=12866, suffix heap size=135192,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=281 2023-04-27 03:54:27,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1682567667269, max timestamp=1682567667276Num uniques 1031; 2023-04-27 03:54:27,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=12866, suffix heap size=135192,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=281 2023-04-27 03:54:27,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1682567667269, max timestamp=1682567667276Num uniques 1031; 2023-04-27 03:54:27,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=12866, suffix heap size=135192,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=281 2023-04-27 03:54:27,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=390, cellSize=17860, totalHeapSize=40800, min timestamp=1682567667307, max timestamp=1682567667315Num uniques 1140; 2023-04-27 03:54:27,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=17860, suffix heap size=146528,new segment heap size=40800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=390 2023-04-27 03:54:27,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=390, cellSize=17860, totalHeapSize=40800, min timestamp=1682567667307, max timestamp=1682567667315Num uniques 1140; 2023-04-27 03:54:27,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=17860, suffix heap size=146528,new segment heap size=40800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=390 2023-04-27 03:54:27,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=390, cellSize=17860, totalHeapSize=40800, min timestamp=1682567667307, max timestamp=1682567667315Num uniques 1140; 2023-04-27 03:54:27,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=17860, suffix heap size=146528,new segment heap size=40800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=390 2023-04-27 03:54:27,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1682567667324, max timestamp=1682567667330Num uniques 1249; 2023-04-27 03:54:27,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=57204, new segment data size=13694, suffix heap size=157864,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1249, new segment cells count=299 2023-04-27 03:54:27,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1682567667324, max timestamp=1682567667330Num uniques 1249; 2023-04-27 03:54:27,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=57204, new segment data size=13694, suffix heap size=157864,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1249, new segment cells count=299 2023-04-27 03:54:27,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1682567667324, max timestamp=1682567667330Num uniques 1249; 2023-04-27 03:54:27,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=57204, new segment data size=13694, suffix heap size=157864,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1249, new segment cells count=299 2023-04-27 03:54:27,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:27,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1682567727351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:27,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:27,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52150 deadline: 1682567727352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:27,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11808, totalHeapSize=27072, min timestamp=1682567667346, max timestamp=1682567667352Num uniques 1158; 2023-04-27 03:54:27,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=53028, new segment data size=11808, suffix heap size=148400,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=258 2023-04-27 03:54:27,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11808, totalHeapSize=27072, min timestamp=1682567667346, max timestamp=1682567667352Num uniques 1158; 2023-04-27 03:54:27,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53028, new segment data size=11808, suffix heap size=148400,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=258 2023-04-27 03:54:27,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11808, totalHeapSize=27072, min timestamp=1682567667346, max timestamp=1682567667352Num uniques 1158; 2023-04-27 03:54:27,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53028, new segment data size=11808, suffix heap size=148400,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=258 2023-04-27 03:54:27,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682567667372, max timestamp=1682567667375Num uniques 1117; 2023-04-27 03:54:27,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-27 03:54:27,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682567667372, max timestamp=1682567667375Num uniques 1117; 2023-04-27 03:54:27,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-27 03:54:27,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682567667372, max timestamp=1682567667375Num uniques 1117; 2023-04-27 03:54:27,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-27 03:54:27,387 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=516, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/16bebecbd3304e91a6b635f47188e041 2023-04-27 03:54:27,401 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/02b11277e5fd4278a11c6159d6234f74 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/02b11277e5fd4278a11c6159d6234f74 2023-04-27 03:54:27,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=426, cellSize=19506, totalHeapSize=44544, min timestamp=1682567667381, max timestamp=1682567667403Num uniques 1076; 2023-04-27 03:54:27,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=19506, suffix heap size=139872,new segment heap size=44544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=426 2023-04-27 03:54:27,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=426, cellSize=19506, totalHeapSize=44544, min timestamp=1682567667381, max timestamp=1682567667403Num uniques 1076; 2023-04-27 03:54:27,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=19506, suffix heap size=139872,new segment heap size=44544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=426 2023-04-27 03:54:27,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=426, cellSize=19506, totalHeapSize=44544, min timestamp=1682567667381, max timestamp=1682567667403Num uniques 1076; 2023-04-27 03:54:27,406 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d66a88af1ba39e44423282d0b046d5b3/A of d66a88af1ba39e44423282d0b046d5b3 into 02b11277e5fd4278a11c6159d6234f74(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:27,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=19506, suffix heap size=139872,new segment heap size=44544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=426 2023-04-27 03:54:27,406 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:27,407 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., storeName=d66a88af1ba39e44423282d0b046d5b3/A, priority=12, startTime=1682567666915; duration=0sec 2023-04-27 03:54:27,407 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:27,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:27,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52164 deadline: 1682567727425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:27,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:27,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1682567727425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:27,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:27,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52168 deadline: 1682567727427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:27,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:27,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1682567727454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:27,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:27,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52150 deadline: 1682567727454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:27,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:27,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:27,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52164 deadline: 1682567727528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:27,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1682567727529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:27,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:27,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52168 deadline: 1682567727531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:27,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:27,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1682567727657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:27,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:27,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52150 deadline: 1682567727658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:27,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:27,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1682567727732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:27,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:27,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:27,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52164 deadline: 1682567727732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:27,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52168 deadline: 1682567727733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:27,800 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/c37fbec62e7d4702bdba9317dec2d990 2023-04-27 03:54:27,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:27,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1682567727960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:27,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:27,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52150 deadline: 1682567727961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:28,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1682567728037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:28,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52164 deadline: 1682567728038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:28,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52168 deadline: 1682567728039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,210 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/6d6f68d70be44fa3bfdc1d3afb68c7ab 2023-04-27 03:54:28,214 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/16bebecbd3304e91a6b635f47188e041 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/16bebecbd3304e91a6b635f47188e041 2023-04-27 03:54:28,220 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/16bebecbd3304e91a6b635f47188e041, entries=100, sequenceid=516, filesize=18.6 K 2023-04-27 03:54:28,221 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/c37fbec62e7d4702bdba9317dec2d990 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/c37fbec62e7d4702bdba9317dec2d990 2023-04-27 03:54:28,225 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/c37fbec62e7d4702bdba9317dec2d990, entries=100, sequenceid=516, filesize=9.5 K 2023-04-27 03:54:28,226 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/6d6f68d70be44fa3bfdc1d3afb68c7ab as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/6d6f68d70be44fa3bfdc1d3afb68c7ab 2023-04-27 03:54:28,230 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/6d6f68d70be44fa3bfdc1d3afb68c7ab, entries=100, sequenceid=516, filesize=9.5 K 2023-04-27 03:54:28,231 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=167.72 KB/171750 for d66a88af1ba39e44423282d0b046d5b3 in 1307ms, sequenceid=516, compaction requested=false 2023-04-27 03:54:28,231 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:28,231 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:28,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:28,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:28,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:28,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:28,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:28,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:28,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:28,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:28,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:28,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:28,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:28,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@155d44c4[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:28,235 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:28,235 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,237 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,237 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:28,238 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:28,238 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:28,238 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,238 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:28,239 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:28,239 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,239 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:28,239 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:28,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:28,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:28,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:28,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:28,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:28,242 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:28,242 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:28,242 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:28,242 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:28,242 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:28,243 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,243 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:28,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:28,243 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:28,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:28,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:28,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:28,244 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:28,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,245 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:28,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,248 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,248 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:28,248 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,248 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:28,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:28,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:28,248 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:28,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:28,248 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:28,248 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:28,248 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:28,248 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,248 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,248 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,248 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:28,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:28,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:28,348 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:28,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:28,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:28,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:28,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:28,353 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:28,353 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:28,353 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:28,353 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:28,354 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,354 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,356 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:28,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:28,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:28,357 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,357 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:28,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:28,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:28,358 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:28,358 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:28,358 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:28,358 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:28,358 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:28,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:28,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:28,360 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:28,360 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:28,361 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d66a88af1ba39e44423282d0b046d5b3 3/3 column families, dataSize=167.72 KB heapSize=459.56 KB 2023-04-27 03:54:28,361 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=A 2023-04-27 03:54:28,361 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:28,361 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=B 2023-04-27 03:54:28,361 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:28,361 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=C 2023-04-27 03:54:28,361 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:28,377 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042710ae3bc1a8b84da6ba1b61a23f4c1d68 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042710ae3bc1a8b84da6ba1b61a23f4c1d68 2023-04-27 03:54:28,383 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=779, memsize=55.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/74cb8aa4afb1460787e59a4f3be5f796 2023-04-27 03:54:28,395 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=779 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/8b083bbd20e54bc4878f0bfd78f46633 2023-04-27 03:54:28,406 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=779 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/1cf0bbd249164846abb885641ba00fe9 2023-04-27 03:54:28,411 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/74cb8aa4afb1460787e59a4f3be5f796 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/74cb8aa4afb1460787e59a4f3be5f796 2023-04-27 03:54:28,415 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/74cb8aa4afb1460787e59a4f3be5f796, entries=150, sequenceid=779, filesize=25.4 K 2023-04-27 03:54:28,416 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/8b083bbd20e54bc4878f0bfd78f46633 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/8b083bbd20e54bc4878f0bfd78f46633 2023-04-27 03:54:28,419 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/8b083bbd20e54bc4878f0bfd78f46633, entries=150, sequenceid=779, filesize=11.9 K 2023-04-27 03:54:28,420 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/1cf0bbd249164846abb885641ba00fe9 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/1cf0bbd249164846abb885641ba00fe9 2023-04-27 03:54:28,425 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/1cf0bbd249164846abb885641ba00fe9, entries=150, sequenceid=779, filesize=11.9 K 2023-04-27 03:54:28,426 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~167.72 KB/171750, heapSize ~459.52 KB/470544, currentSize=0 B/0 for d66a88af1ba39e44423282d0b046d5b3 in 66ms, sequenceid=779, compaction requested=true 2023-04-27 03:54:28,426 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:28,426 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:28,426 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:28,427 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:28,427 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,428 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:28,428 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:28,429 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,429 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:28,430 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,430 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:28,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:28,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17be18e9[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:28,432 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:28,432 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,433 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,433 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:28,433 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:28,433 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:28,433 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,434 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:28,435 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,435 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:28,435 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:28,435 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:28,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:28,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:28,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:28,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:28,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:28,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:28,438 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:28,438 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:28,438 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:28,438 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:28,438 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:28,441 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,441 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:28,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,441 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:28,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:28,442 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:28,443 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,445 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:28,445 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:28,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:28,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:28,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:28,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:28,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:28,445 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:28,445 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:28,446 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,446 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,446 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,446 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:28,446 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,446 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:28,446 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,446 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:28,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:28,500 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d66a88af1ba39e44423282d0b046d5b3 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:54:28,501 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=A 2023-04-27 03:54:28,501 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:28,501 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=B 2023-04-27 03:54:28,501 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:28,501 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=C 2023-04-27 03:54:28,501 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:28,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682567668519, max timestamp=1682567668538Num uniques 859; 2023-04-27 03:54:28,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-27 03:54:28,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682567668519, max timestamp=1682567668538Num uniques 859; 2023-04-27 03:54:28,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682567668519, max timestamp=1682567668538Num uniques 859; 2023-04-27 03:54:28,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-27 03:54:28,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-27 03:54:28,547 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:28,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:28,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:28,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:28,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:28,550 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:28,550 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:28,550 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:28,551 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:28,551 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,551 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,555 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:28,555 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:28,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:28,555 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,555 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:28,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:28,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:28,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:28,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:28,556 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:28,557 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:28,557 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:28,557 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:28,557 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:28,557 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:28,557 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:28,557 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:28,557 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:28,557 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:28,557 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:28,557 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,557 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:28,557 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:28,559 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:28,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1682567728555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,559 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,559 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:28,560 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,560 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:28,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:28,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@623e6852[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:28,561 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:28,561 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:28,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52168 deadline: 1682567728558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:28,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52164 deadline: 1682567728559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,562 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:28,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52150 deadline: 1682567728559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:28,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:28,563 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,563 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:28,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1682567728559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:28,564 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,564 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:28,564 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:28,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:28,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:28,564 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:28,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:28,566 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:28,566 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:28,567 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:28,567 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:28,567 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:28,568 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,568 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,568 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:28,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:28,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:28,569 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,570 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,574 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:28,574 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:28,574 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:28,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:28,574 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:28,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:28,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:28,574 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:28,575 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,575 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:28,575 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:28,575 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,575 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:28,575 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,575 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:28,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1682567728660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:28,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52168 deadline: 1682567728662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:28,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52164 deadline: 1682567728663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:28,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52150 deadline: 1682567728663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:28,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1682567728664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,675 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:28,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:28,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:28,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:28,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:28,678 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:28,678 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:28,678 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:28,678 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:28,678 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,678 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,680 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,680 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:28,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:28,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:28,680 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,680 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:28,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:28,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:28,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:28,681 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:28,681 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:28,681 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:28,681 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:28,681 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:28,681 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:28,681 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:28,682 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:28,682 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:28,682 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:28,682 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:28,682 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,682 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:28,682 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:28,683 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,683 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:28,684 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,684 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:28,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:28,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60eead6e[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:28,685 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:28,685 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,690 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,690 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:28,690 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,690 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:28,690 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:28,690 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:28,692 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,692 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:28,693 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:28,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:28,693 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:28,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:28,695 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:28,695 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:28,695 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:28,695 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:28,695 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:28,697 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,697 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:28,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,697 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:28,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:28,697 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,698 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:28,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,700 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,700 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,700 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,700 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,700 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:28,700 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,701 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:28,701 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:28,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:28,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:28,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:28,701 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:28,701 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:28,701 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,701 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:28,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:28,801 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:28,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:28,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:28,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:28,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:28,805 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:28,805 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:28,805 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:28,805 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:28,805 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,805 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,806 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:28,806 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:28,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:28,807 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,807 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:28,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:28,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:28,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:28,808 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:28,808 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:28,808 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:28,808 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:28,808 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:28,808 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:28,808 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:28,808 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:28,808 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:28,809 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:28,809 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:28,809 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,809 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:28,809 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:28,811 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,811 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:28,811 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,811 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:28,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:28,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f6aaed3[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:28,813 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:28,813 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,814 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,814 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:28,814 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:28,814 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:28,814 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,814 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:28,816 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:28,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:28,818 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:28,818 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:28,818 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:28,818 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:28,818 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:28,820 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,820 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,820 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:28,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:28,820 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,822 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,826 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,826 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,826 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,826 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:28,826 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:28,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:28,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:28,826 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,826 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:28,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:28,826 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,826 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:28,826 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:28,826 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,826 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:28,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:28,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52168 deadline: 1682567728864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:28,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1682567728864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:28,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52164 deadline: 1682567728865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:28,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52150 deadline: 1682567728865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:28,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1682567728868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,919 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304274735a865f21e4c3894289d854983beed to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304274735a865f21e4c3894289d854983beed 2023-04-27 03:54:28,926 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:28,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:28,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:28,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:28,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:28,929 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:28,929 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:28,929 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:28,929 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:28,930 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,930 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,931 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:28,931 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:28,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:28,931 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,931 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:28,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:28,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:28,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:28,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:28,933 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:28,933 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:28,933 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:28,934 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:28,934 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:28,934 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:28,934 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:28,934 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:28,934 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:28,934 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:28,934 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:28,934 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,934 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:28,934 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,936 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:28,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:28,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:28,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:28,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78f48487[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:28,938 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:28,938 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,942 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,942 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:28,942 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,942 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:28,942 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:28,942 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:28,944 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:28,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:28,947 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:28,947 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:28,947 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:28,947 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:28,947 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:28,948 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,948 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,948 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:28,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:28,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:28,949 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,950 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,952 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,952 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,952 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:28,952 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:28,952 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:28,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:28,952 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:28,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:28,953 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:28,953 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:28,953 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:28,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:28,953 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:28,953 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:28,953 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:28,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:29,053 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:29,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:29,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:29,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:29,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:29,057 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:29,057 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:29,057 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:29,057 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:29,058 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,058 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,060 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,060 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:29,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:29,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:29,061 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,061 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:29,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:29,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,061 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:29,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:29,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:29,062 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:29,062 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:29,062 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:29,062 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:29,062 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:29,062 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:29,062 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:29,062 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:29,062 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:29,062 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:29,062 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:29,062 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,062 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:29,062 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:29,064 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:29,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:29,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:29,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69991f33[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:29,065 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:29,065 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,067 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,067 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:29,067 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:29,067 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:29,067 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,067 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:29,068 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,068 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:29,068 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:29,068 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:29,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:29,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:29,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:29,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:29,070 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:29,070 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:29,070 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:29,070 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:29,070 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:29,072 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,072 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,072 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:29,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,073 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:29,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,075 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,077 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,077 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,077 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,077 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:29,077 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:29,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:29,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:29,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:29,077 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:29,077 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:29,077 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,077 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,077 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:29,077 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,077 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:29,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:29,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:29,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:29,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52168 deadline: 1682567729169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52150 deadline: 1682567729170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:29,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:29,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1682567729170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1682567729170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:29,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52164 deadline: 1682567729171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,177 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:29,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:29,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:29,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:29,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:29,181 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:29,181 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:29,181 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:29,181 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:29,181 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,181 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,183 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,183 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:29,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:29,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:29,183 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,183 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:29,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:29,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:29,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:29,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:29,184 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:29,184 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:29,184 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:29,184 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:29,184 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:29,184 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:29,184 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:29,184 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:29,185 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:29,185 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:29,185 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:29,185 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,185 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:29,185 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:29,186 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,186 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:29,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:29,186 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,186 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:29,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:29,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c895ab3[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:29,188 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:29,188 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,189 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,190 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:29,190 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:29,190 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:29,190 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,190 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:29,191 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:29,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:29,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:29,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:29,193 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:29,193 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:29,193 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:29,193 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:29,193 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:29,195 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,195 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,195 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:29,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:29,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:29,195 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:29,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,197 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,199 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:29,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:29,199 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,199 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:29,199 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,199 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,199 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:29,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:29,199 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:29,199 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,200 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,200 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,200 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:29,200 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:29,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:29,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:29,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:29,300 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:29,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:29,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:29,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:29,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:29,303 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:29,303 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:29,303 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:29,303 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:29,304 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,304 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,306 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:29,306 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:29,307 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,307 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:29,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:29,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:29,309 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:29,309 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:29,309 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:29,309 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:29,309 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:29,309 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:29,309 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:29,309 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:29,309 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:29,309 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:29,309 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:29,309 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,309 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:29,309 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:29,311 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,311 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:29,311 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,311 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:29,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:29,313 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:29,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36df7fc7[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:29,313 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,314 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,314 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:29,314 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:29,314 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:29,314 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,315 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:29,318 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:29,318 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,319 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:29,319 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:29,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:29,321 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:29,321 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:29,321 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:29,321 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:29,321 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:29,322 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,322 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:29,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,322 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:29,323 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:29,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,324 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,325 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=790, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/d19b3a7d39594026a2957e24f9b53b03 2023-04-27 03:54:29,326 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:29,326 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:29,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:29,326 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:29,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:29,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:29,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:29,327 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:29,327 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:29,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:29,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:29,327 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,327 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,327 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,327 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,327 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,327 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,327 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,338 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=790 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/4c1512a9fba34205bf52da4cccc6b400 2023-04-27 03:54:29,427 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:29,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:29,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:29,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:29,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:29,430 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:29,430 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:29,430 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:29,430 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:29,431 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,431 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,432 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:29,432 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:29,432 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,432 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:29,433 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:29,433 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:29,434 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:29,434 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:29,434 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:29,434 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:29,434 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:29,434 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:29,434 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:29,434 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:29,434 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:29,434 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:29,434 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:29,434 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,434 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:29,434 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:29,437 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,437 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:29,437 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,437 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:29,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:29,438 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:29,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a531527[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:29,438 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,439 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:29,440 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,440 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:29,441 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:29,441 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:29,441 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:29,442 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:29,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:29,444 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:29,444 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:29,444 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:29,444 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:29,444 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:29,446 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,446 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:29,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:29,446 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:29,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:29,446 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:29,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:29,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:29,448 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,451 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:29,451 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:29,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:29,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:29,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:29,451 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:29,451 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,452 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,452 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,452 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,452 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,452 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:29,452 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:29,452 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,452 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:29,552 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:29,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:29,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:29,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:29,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:29,555 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:29,555 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:29,555 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:29,555 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:29,556 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,556 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,557 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:29,557 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:29,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:29,557 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,557 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:29,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:29,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:29,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:29,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:29,559 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:29,559 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:29,559 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:29,559 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:29,559 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:29,559 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:29,559 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:29,559 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:29,559 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:29,559 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:29,559 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:29,559 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,559 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:29,559 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:29,561 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,561 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:29,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:29,561 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,561 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:29,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:29,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:29,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:29,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b12ffd9[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:29,562 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:29,562 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,564 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,564 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:29,564 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,564 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:29,564 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:29,564 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:29,566 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,566 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:29,566 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:29,567 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:29,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:29,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:29,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:29,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:29,569 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:29,569 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:29,569 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:29,569 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:29,569 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:29,570 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,570 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:29,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:29,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,570 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:29,571 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:29,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,572 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,574 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,574 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,574 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:29,574 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:29,574 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:29,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:29,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:29,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:29,574 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:29,574 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:29,574 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:29,574 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,575 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:29,575 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,575 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:29,675 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:29,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:29,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:29,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1682567729673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:29,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52164 deadline: 1682567729674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:29,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52168 deadline: 1682567729675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:29,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:29,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1682567729676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:29,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52150 deadline: 1682567729677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:29,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:29,678 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:29,678 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:29,678 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:29,678 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:29,679 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,679 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,680 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,680 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:29,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:29,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:29,681 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,681 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:29,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:29,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:29,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:29,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:29,682 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:29,682 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:29,682 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:29,682 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:29,682 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:29,682 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:29,682 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:29,682 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:29,682 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:29,682 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:29,682 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:29,682 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,683 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:29,683 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:29,685 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,685 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:29,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:29,685 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,685 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:29,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:29,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:29,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:29,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:29,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d7868ca[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:29,687 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:29,687 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,688 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,688 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:29,688 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:29,688 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:29,688 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,688 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:29,690 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:29,690 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:29,690 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,690 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:29,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:29,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:29,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:29,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:29,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:29,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:29,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:29,692 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:29,692 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:29,692 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:29,692 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:29,692 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:29,693 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,694 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:29,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,694 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:29,694 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,695 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,702 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:29,702 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:29,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:29,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:29,702 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,702 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:29,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:29,702 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:29,702 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:29,702 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:29,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:29,702 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,702 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,702 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,702 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:29,702 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:29,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:29,748 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=790 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/6a1d840517fe4bde831ee8cde17059fa 2023-04-27 03:54:29,752 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/d19b3a7d39594026a2957e24f9b53b03 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/d19b3a7d39594026a2957e24f9b53b03 2023-04-27 03:54:29,756 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/d19b3a7d39594026a2957e24f9b53b03, entries=400, sequenceid=790, filesize=59.8 K 2023-04-27 03:54:29,757 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/4c1512a9fba34205bf52da4cccc6b400 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/4c1512a9fba34205bf52da4cccc6b400 2023-04-27 03:54:29,760 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/4c1512a9fba34205bf52da4cccc6b400, entries=150, sequenceid=790, filesize=11.9 K 2023-04-27 03:54:29,761 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/6a1d840517fe4bde831ee8cde17059fa as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/6a1d840517fe4bde831ee8cde17059fa 2023-04-27 03:54:29,764 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/6a1d840517fe4bde831ee8cde17059fa, entries=150, sequenceid=790, filesize=11.9 K 2023-04-27 03:54:29,765 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=147.60 KB/151140 for d66a88af1ba39e44423282d0b046d5b3 in 1265ms, sequenceid=790, compaction requested=true 2023-04-27 03:54:29,765 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:29,765 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:29,765 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:29,765 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 03:54:29,765 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:29,765 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 03:54:29,767 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 133060 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 03:54:29,767 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): d66a88af1ba39e44423282d0b046d5b3/A is initiating minor compaction (all files) 2023-04-27 03:54:29,767 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47016 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 03:54:29,767 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d66a88af1ba39e44423282d0b046d5b3/A in TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:29,767 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): d66a88af1ba39e44423282d0b046d5b3/B is initiating minor compaction (all files) 2023-04-27 03:54:29,767 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/02b11277e5fd4278a11c6159d6234f74, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/16bebecbd3304e91a6b635f47188e041, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/74cb8aa4afb1460787e59a4f3be5f796, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/d19b3a7d39594026a2957e24f9b53b03] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp, totalSize=129.9 K 2023-04-27 03:54:29,767 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d66a88af1ba39e44423282d0b046d5b3/B in TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:29,767 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/83116b4a1eaa4231bab25fef8f75081f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/c37fbec62e7d4702bdba9317dec2d990, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/8b083bbd20e54bc4878f0bfd78f46633, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/4c1512a9fba34205bf52da4cccc6b400] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp, totalSize=45.9 K 2023-04-27 03:54:29,767 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 02b11277e5fd4278a11c6159d6234f74, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1682567664866 2023-04-27 03:54:29,768 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 83116b4a1eaa4231bab25fef8f75081f, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1682567664866 2023-04-27 03:54:29,768 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 16bebecbd3304e91a6b635f47188e041, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1682567665995 2023-04-27 03:54:29,768 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting c37fbec62e7d4702bdba9317dec2d990, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1682567665995 2023-04-27 03:54:29,769 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 74cb8aa4afb1460787e59a4f3be5f796, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=779, earliestPutTs=1682567667424 2023-04-27 03:54:29,769 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 8b083bbd20e54bc4878f0bfd78f46633, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=779, earliestPutTs=1682567667424 2023-04-27 03:54:29,769 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting d19b3a7d39594026a2957e24f9b53b03, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=790, earliestPutTs=1682567668486 2023-04-27 03:54:29,769 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 4c1512a9fba34205bf52da4cccc6b400, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=790, earliestPutTs=1682567668492 2023-04-27 03:54:29,778 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): d66a88af1ba39e44423282d0b046d5b3#B#compaction#188 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:29,779 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): d66a88af1ba39e44423282d0b046d5b3#A#compaction#187 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:29,797 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/66ad8e143e4b424ea32e0f1c1d5ad4d2 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/66ad8e143e4b424ea32e0f1c1d5ad4d2 2023-04-27 03:54:29,801 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d66a88af1ba39e44423282d0b046d5b3/B of d66a88af1ba39e44423282d0b046d5b3 into 66ad8e143e4b424ea32e0f1c1d5ad4d2(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:29,801 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:29,801 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., storeName=d66a88af1ba39e44423282d0b046d5b3/B, priority=12, startTime=1682567669765; duration=0sec 2023-04-27 03:54:29,801 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:29,801 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 03:54:29,802 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:29,802 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46982 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 03:54:29,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:29,803 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): d66a88af1ba39e44423282d0b046d5b3/C is initiating minor compaction (all files) 2023-04-27 03:54:29,803 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d66a88af1ba39e44423282d0b046d5b3/C in TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:29,803 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/122250d39b384ab29709f768e0f1dfa0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/6d6f68d70be44fa3bfdc1d3afb68c7ab, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/1cf0bbd249164846abb885641ba00fe9, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/6a1d840517fe4bde831ee8cde17059fa] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp, totalSize=45.9 K 2023-04-27 03:54:29,804 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 122250d39b384ab29709f768e0f1dfa0, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1682567664866 2023-04-27 03:54:29,804 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 6d6f68d70be44fa3bfdc1d3afb68c7ab, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1682567665995 2023-04-27 03:54:29,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:29,805 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 1cf0bbd249164846abb885641ba00fe9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=779, earliestPutTs=1682567667424 2023-04-27 03:54:29,805 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 6a1d840517fe4bde831ee8cde17059fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=790, earliestPutTs=1682567668492 2023-04-27 03:54:29,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:29,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:29,816 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:29,816 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:29,816 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:29,816 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:29,816 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,817 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,818 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:29,818 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:29,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:29,818 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): d66a88af1ba39e44423282d0b046d5b3#C#compaction#189 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:29,818 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:29,818 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:29,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:29,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:29,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:29,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:29,820 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:29,820 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:29,820 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:29,820 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:29,820 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:29,824 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:29,824 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:29,824 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:29,824 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:29,824 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d66a88af1ba39e44423282d0b046d5b3 3/3 column families, dataSize=147.60 KB heapSize=410.77 KB 2023-04-27 03:54:29,824 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=A 2023-04-27 03:54:29,824 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:29,824 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=B 2023-04-27 03:54:29,825 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:29,825 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=C 2023-04-27 03:54:29,825 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:29,846 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/61b46a9a1b474d4bbde8a3f2ee2c35fa as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/61b46a9a1b474d4bbde8a3f2ee2c35fa 2023-04-27 03:54:29,856 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d66a88af1ba39e44423282d0b046d5b3/C of d66a88af1ba39e44423282d0b046d5b3 into 61b46a9a1b474d4bbde8a3f2ee2c35fa(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:29,856 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:29,856 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., storeName=d66a88af1ba39e44423282d0b046d5b3/C, priority=12, startTime=1682567669765; duration=0sec 2023-04-27 03:54:29,856 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:29,865 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304273019652855b44597bbff8fbd7de9791d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304273019652855b44597bbff8fbd7de9791d 2023-04-27 03:54:29,870 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=827, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/0b46013496bf4d099a098ead819001d9 2023-04-27 03:54:30,279 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=827 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/4024ac7bebfd4e9ba6fbd4b7d5d7a074 2023-04-27 03:54:30,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:30,686 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:30,691 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=827 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/8bd0a267ad58425190608d660acfb318 2023-04-27 03:54:30,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:30,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52164 deadline: 1682567730687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:30,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1682567730688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:30,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52168 deadline: 1682567730688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:30,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1682567730689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,693 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:30,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52150 deadline: 1682567730692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,696 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/0b46013496bf4d099a098ead819001d9 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/0b46013496bf4d099a098ead819001d9 2023-04-27 03:54:30,700 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/0b46013496bf4d099a098ead819001d9, entries=150, sequenceid=827, filesize=25.4 K 2023-04-27 03:54:30,700 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/4024ac7bebfd4e9ba6fbd4b7d5d7a074 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/4024ac7bebfd4e9ba6fbd4b7d5d7a074 2023-04-27 03:54:30,707 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/4024ac7bebfd4e9ba6fbd4b7d5d7a074, entries=150, sequenceid=827, filesize=11.9 K 2023-04-27 03:54:30,707 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/8bd0a267ad58425190608d660acfb318 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/8bd0a267ad58425190608d660acfb318 2023-04-27 03:54:30,711 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/8bd0a267ad58425190608d660acfb318, entries=150, sequenceid=827, filesize=11.9 K 2023-04-27 03:54:30,712 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=40.25 KB/41220 for d66a88af1ba39e44423282d0b046d5b3 in 888ms, sequenceid=827, compaction requested=false 2023-04-27 03:54:30,712 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:30,712 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:30,712 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:30,712 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:30,712 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,712 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:30,712 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:30,714 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:30,714 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:30,715 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:30,715 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:30,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:30,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@309401e5[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:30,717 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:30,717 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:30,718 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:30,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:30,719 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:30,719 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:30,719 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:30,719 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,719 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:30,721 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:30,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:30,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:30,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:30,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:30,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:30,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:30,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:30,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:30,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:30,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:30,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:30,724 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:30,724 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:30,724 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:30,724 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:30,724 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:30,726 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,726 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:30,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,726 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:30,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:30,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:30,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:30,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:30,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:30,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:30,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:30,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:30,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:30,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,732 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,732 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:30,732 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:30,732 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:30,732 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:30,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:30,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:30,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:30,732 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,733 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:30,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:30,733 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:30,733 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:30,733 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:30,733 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:30,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:30,733 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:30,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:30,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:30,795 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d66a88af1ba39e44423282d0b046d5b3 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:54:30,795 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=A 2023-04-27 03:54:30,795 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:30,795 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=B 2023-04-27 03:54:30,795 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:30,796 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=C 2023-04-27 03:54:30,796 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:30,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682567670813, max timestamp=1682567670816Num uniques 859; 2023-04-27 03:54:30,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:54:30,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682567670813, max timestamp=1682567670816Num uniques 859; 2023-04-27 03:54:30,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682567670813, max timestamp=1682567670816Num uniques 859; 2023-04-27 03:54:30,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:54:30,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:54:30,833 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:30,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:30,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:30,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52150 deadline: 1682567730829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:30,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52168 deadline: 1682567730830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:30,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:30,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1682567730832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:30,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1682567730833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:30,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:30,836 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:30,836 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:30,836 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:30,836 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:30,837 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,837 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:30,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:30,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52164 deadline: 1682567730834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,838 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:30,838 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:30,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:30,838 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,838 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:30,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:30,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:30,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:30,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:30,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:30,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:30,839 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:30,839 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:30,839 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:30,839 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:30,839 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:30,840 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:30,840 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:30,840 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:30,840 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:30,840 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:30,840 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:30,840 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,840 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:30,840 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:30,843 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,843 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:30,843 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:30,843 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:30,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:30,844 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:30,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@429cc0fd[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:30,844 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:30,846 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:30,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:30,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:30,846 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:30,846 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,846 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:30,846 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:30,846 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:30,847 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,847 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:30,847 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:30,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:30,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:30,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:30,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:30,849 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:30,849 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:30,850 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:30,850 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:30,850 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:30,851 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,851 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:30,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:30,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,851 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:30,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:30,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:30,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:30,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:30,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:30,853 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,855 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,855 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:30,855 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:30,856 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:30,856 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:30,856 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:30,856 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,856 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:30,856 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:30,856 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:30,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:30,856 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:30,856 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:30,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:30,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:30,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:30,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:30,937 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:30,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52150 deadline: 1682567730935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:30,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52168 deadline: 1682567730935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:30,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1682567730936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:30,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1682567730936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:30,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52164 deadline: 1682567730939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,956 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:30,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:30,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:30,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:30,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:30,959 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:30,959 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:30,960 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:30,960 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:30,960 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,960 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:30,961 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:30,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:30,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:30,961 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,962 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,962 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:30,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:30,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:30,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:30,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:30,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:30,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:30,963 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:30,963 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:30,963 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:30,963 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:30,963 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:30,963 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:30,963 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:30,963 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:30,963 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:30,963 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:30,964 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:30,964 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,964 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:30,964 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:30,965 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:30,965 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:30,965 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:30,965 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:30,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:30,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eac1340[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:30,967 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:30,967 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:30,968 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:30,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:30,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:30,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:30,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:30,969 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,969 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:30,974 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,974 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:30,974 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:30,974 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:30,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:30,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:30,976 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:30,976 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:30,976 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:30,976 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:30,976 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:30,978 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:30,978 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,978 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:30,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:30,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:30,978 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,980 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,984 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,984 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:30,984 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:30,984 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:30,984 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:30,984 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:30,984 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:30,985 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:30,985 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:30,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:30,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:30,985 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:30,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:30,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:30,985 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:30,985 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:30,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:30,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:31,005 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/4138f95cbc5948b2ba5b1c76d2011538 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/4138f95cbc5948b2ba5b1c76d2011538 2023-04-27 03:54:31,009 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d66a88af1ba39e44423282d0b046d5b3/A of d66a88af1ba39e44423282d0b046d5b3 into 4138f95cbc5948b2ba5b1c76d2011538(size=26.3 K), total size for store is 51.7 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-27 03:54:31,009 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:31,009 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., storeName=d66a88af1ba39e44423282d0b046d5b3/A, priority=12, startTime=1682567669765; duration=1sec 2023-04-27 03:54:31,009 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:31,085 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:31,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:31,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:31,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:31,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:31,088 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:31,088 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:31,088 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:31,088 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:31,089 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,089 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,090 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,090 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:31,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:31,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:31,090 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,090 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:31,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:31,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,091 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:31,091 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:31,091 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:31,091 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:31,091 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:31,091 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:31,091 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:31,091 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:31,091 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:31,091 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:31,092 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:31,092 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:31,092 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:31,092 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:31,092 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,092 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:31,092 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:31,093 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,093 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:31,093 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,093 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:31,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:31,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:31,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:31,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:31,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@202c0667[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:31,095 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:31,095 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,097 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,097 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,098 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:31,097 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:31,098 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:31,098 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:31,099 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,099 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:31,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,099 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:31,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:31,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:31,099 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:31,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:31,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:31,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:31,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:31,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:31,101 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:31,101 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:31,101 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:31,101 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:31,101 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:31,103 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,103 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:31,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:31,103 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:31,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:31,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:31,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:31,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,104 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:31,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:31,105 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,107 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,107 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,107 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,107 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:31,107 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:31,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:31,107 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:31,107 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:31,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:31,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:31,108 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,108 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,108 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,108 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:31,108 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:31,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:31,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:31,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52150 deadline: 1682567731139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:31,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52168 deadline: 1682567731139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:31,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1682567731139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:31,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1682567731139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:31,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52164 deadline: 1682567731142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,208 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:31,209 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427fd4064b1204741b0abd30d0034f92c53 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427fd4064b1204741b0abd30d0034f92c53 2023-04-27 03:54:31,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:31,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:31,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:31,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:31,212 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:31,212 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:31,213 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:31,213 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:31,213 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,213 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,215 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,215 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:31,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:31,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:31,215 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,215 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:31,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:31,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,216 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:31,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:31,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:31,216 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:31,216 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:31,216 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:31,216 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:31,217 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:31,217 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:31,217 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:31,217 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:31,217 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:31,217 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:31,217 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:31,217 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,217 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:31,217 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:31,218 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,218 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:31,219 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,219 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:31,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:31,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30a7b0ae[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:31,220 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:31,221 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,222 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,222 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:31,222 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:31,222 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,222 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:31,222 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:31,223 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,223 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:31,224 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:31,224 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:31,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:31,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:31,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:31,226 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:31,226 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:31,226 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:31,226 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:31,226 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:31,229 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,229 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:31,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,229 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:31,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:31,230 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,231 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,234 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,234 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,234 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,234 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:31,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:31,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:31,234 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,234 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:31,234 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:31,234 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:31,234 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,234 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:31,234 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:31,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:31,334 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:31,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:31,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:31,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:31,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:31,338 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:31,338 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:31,338 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:31,338 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:31,338 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,338 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,339 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:31,339 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:31,340 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,340 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:31,341 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:31,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:31,341 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:31,341 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:31,341 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:31,341 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:31,341 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:31,341 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:31,341 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:31,341 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:31,341 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:31,341 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:31,341 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:31,341 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,341 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:31,341 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:31,343 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,343 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:31,343 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,343 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:31,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:31,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:31,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:31,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1498056f[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:31,345 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:31,345 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,347 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,347 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,347 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:31,347 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:31,347 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:31,347 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:31,349 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,349 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:31,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:31,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:31,349 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:31,349 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:31,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:31,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:31,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:31,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:31,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:31,351 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:31,351 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:31,351 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:31,351 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:31,351 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:31,353 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,353 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:31,353 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:31,353 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:31,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,354 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:31,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,356 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,356 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,356 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,356 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:31,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:31,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:31,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:31,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:31,356 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:31,356 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:31,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:31,357 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,357 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,357 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,357 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:31,357 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:31,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:31,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:31,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:31,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52150 deadline: 1682567731441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:31,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1682567731442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:31,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1682567731443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:31,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52168 deadline: 1682567731444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:31,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52164 deadline: 1682567731445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,457 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:31,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:31,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:31,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:31,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:31,460 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:31,460 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:31,460 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:31,460 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:31,461 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,461 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,465 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,465 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:31,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:31,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:31,465 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,466 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:31,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:31,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:31,467 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:31,467 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:31,467 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:31,467 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:31,467 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:31,467 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:31,467 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:31,467 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:31,467 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:31,467 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:31,467 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:31,467 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:31,467 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:31,467 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,467 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:31,467 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:31,469 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,469 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:31,469 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,469 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:31,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:31,470 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:31,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a5032af[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:31,471 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,472 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:31,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:31,472 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,472 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:31,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:31,475 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:31,475 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,475 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:31,475 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:31,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:31,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:31,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:31,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:31,477 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:31,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:31,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:31,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:31,477 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:31,479 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,479 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:31,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:31,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,479 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:31,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:31,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:31,479 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:31,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:31,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,481 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:31,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,483 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,483 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:31,483 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:31,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:31,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:31,483 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:31,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:31,483 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:31,483 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:31,483 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:31,483 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,483 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:31,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:31,483 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:31,483 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,583 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:31,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:31,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:31,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:31,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:31,587 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:31,587 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:31,587 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:31,587 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:31,587 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,587 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,588 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,588 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:31,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:31,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:31,589 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,589 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:31,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:31,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,590 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:31,590 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:31,590 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:31,590 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:31,590 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:31,590 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:31,590 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:31,590 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:31,590 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:31,590 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:31,591 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:31,591 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:31,591 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:31,591 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:31,591 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,591 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:31,591 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:31,593 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,593 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:31,594 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,594 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:31,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:31,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:31,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:31,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:31,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a0f1ebc[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:31,595 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:31,595 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,596 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:31,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:31,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:31,597 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,597 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:31,599 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:31,599 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:31,599 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,599 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:31,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:31,601 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:31,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:31,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:31,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:31,601 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:31,603 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,603 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:31,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,603 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:31,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:31,603 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,604 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,608 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,608 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:31,608 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:31,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:31,608 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:31,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:31,608 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:31,608 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:31,608 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,608 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:31,609 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,609 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:31,609 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,609 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:31,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:31,615 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=838, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/b3c0376b729641239275013377660973 2023-04-27 03:54:31,709 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:31,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:31,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:31,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:31,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:31,712 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:31,712 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:31,712 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:31,712 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:31,713 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,713 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,714 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,714 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:31,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:31,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:31,714 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,714 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:31,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:31,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,715 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:31,716 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:31,716 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:31,716 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:31,716 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:31,716 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:31,716 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:31,716 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:31,716 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:31,716 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:31,716 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:31,716 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:31,716 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:31,716 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:31,716 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,716 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:31,716 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:31,718 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,718 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:31,718 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:31,718 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:31,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:31,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2720f993[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:31,719 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:31,720 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,721 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:31,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:31,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:31,721 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,721 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:31,723 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:31,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:31,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:31,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:31,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:31,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:31,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:31,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:31,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:31,726 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:31,726 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:31,726 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:31,726 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:31,726 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:31,728 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,728 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:31,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,728 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:31,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:31,728 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:31,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:31,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,732 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,732 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:31,732 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:31,732 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,732 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:31,732 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:31,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:31,732 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:31,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:31,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:31,732 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,732 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:31,732 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:31,732 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,733 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:31,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:31,833 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:31,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:31,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:31,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:31,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:31,836 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:31,836 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:31,836 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:31,836 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:31,837 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,837 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,838 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:31,838 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:31,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:31,838 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,838 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:31,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:31,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:31,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:31,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:31,840 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:31,840 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:31,840 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:31,840 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:31,840 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:31,840 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:31,840 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:31,840 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:31,840 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:31,840 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:31,840 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:31,840 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,840 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:31,840 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:31,843 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,843 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:31,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:31,843 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,843 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:31,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:31,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:31,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:31,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:31,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73c73365[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:31,844 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:31,845 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,849 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,849 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,849 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:31,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:31,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:31,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:31,851 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:31,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:31,851 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:31,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:31,851 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:31,851 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:31,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:31,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:31,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:31,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:31,853 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:31,853 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:31,853 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:31,853 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:31,853 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:31,855 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,855 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,855 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:31,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:31,855 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:31,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:31,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,856 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:31,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:31,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:31,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:31,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:31,859 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:31,860 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:31,860 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:31,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:31,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:31,860 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,860 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,860 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,860 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:31,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:31,948 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:31,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52150 deadline: 1682567731945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,948 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:31,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52140 deadline: 1682567731945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,948 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:31,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52160 deadline: 1682567731945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,948 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:31,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52168 deadline: 1682567731947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,948 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:31,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52164 deadline: 1682567731948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,960 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:31,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:31,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:31,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:31,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:31,963 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:31,963 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:31,963 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:31,963 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:31,963 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,963 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,965 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,965 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:31,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:31,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:31,965 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,965 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:31,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:31,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,966 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:31,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:31,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:31,966 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:31,966 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:31,966 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:31,966 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:31,966 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:31,966 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:31,966 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:31,966 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:31,966 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:31,967 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:31,967 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:31,967 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,967 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:31,967 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:31,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,968 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:31,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:31,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:31,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:31,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:31,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:31,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:31,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18e9b2f8[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:31,969 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:31,969 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,971 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:31,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:31,971 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,971 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:31,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:31,973 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:31,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:31,973 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:31,974 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:31,974 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:31,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:31,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:31,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:31,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:31,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:31,976 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:31,976 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:31,976 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:31,976 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:31,976 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:31,977 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,977 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:31,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:31,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,977 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:31,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:31,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:31,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:31,978 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:31,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,979 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:31,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:31,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,981 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,981 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:31,981 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:31,981 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:31,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:31,981 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:31,982 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:31,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:31,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:31,982 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:31,982 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,982 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:31,982 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:31,982 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:31,982 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:31,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:31,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:32,026 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=838 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/77bc02333f564c88956b99ed1f86c983 2023-04-27 03:54:32,082 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:32,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:32,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:32,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:32,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:32,085 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:32,085 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:32,085 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:32,085 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:32,086 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,086 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:32,087 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,087 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:32,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:32,087 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,087 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:32,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:32,088 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:32,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:32,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:32,088 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:32,088 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:32,088 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:32,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:32,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:32,089 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:32,089 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:32,089 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:32,089 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:32,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:32,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:32,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:32,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:32,091 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,091 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:32,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:32,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:32,092 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:32,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:32,092 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:32,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:32,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:32,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:32,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:32,093 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:32,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a41a5a7[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:32,093 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:32,094 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:32,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:32,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:32,094 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:32,094 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:32,094 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:32,095 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,095 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:32,096 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,096 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:32,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:32,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:32,096 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:32,096 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:32,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:32,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:32,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:32,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:32,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:32,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:32,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:32,098 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:32,098 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:32,098 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:32,098 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:32,098 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:32,099 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,099 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:32,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:32,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,099 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:32,100 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:32,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:32,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:32,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:32,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:32,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:32,101 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:32,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:32,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,104 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:32,104 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:32,104 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:32,104 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:32,105 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:32,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:32,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:32,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:32,105 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:32,105 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,105 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:32,105 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:32,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:32,105 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:32,105 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,105 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:32,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:32,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:32,205 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:32,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:32,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:32,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:32,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:32,208 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:32,208 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:32,208 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:32,208 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:32,209 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,209 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:32,210 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:32,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:32,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:32,210 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:32,211 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,211 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:32,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:32,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:32,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:32,212 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:32,212 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:32,212 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:32,212 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:32,212 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:32,212 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:32,212 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:32,212 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:32,212 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:32,212 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:32,212 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:32,212 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:32,212 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:32,212 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,212 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:32,212 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:32,213 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,214 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:32,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:32,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:32,214 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:32,214 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:32,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:32,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:32,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:32,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:32,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:32,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a8d311d[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:32,215 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:32,216 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:32,217 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:32,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:32,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:32,217 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:32,217 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:32,217 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:32,217 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,217 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:32,222 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:32,222 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,222 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:32,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:32,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:32,222 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:32,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:32,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:32,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:32,224 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:32,224 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:32,224 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:32,224 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:32,224 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:32,229 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,229 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:32,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:32,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,229 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:32,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:32,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:32,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:32,229 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:32,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:32,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:32,230 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:32,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:32,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,238 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:32,238 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:32,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:32,239 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:32,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:32,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:32,238 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:32,238 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:32,239 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:32,239 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,239 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:32,239 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:32,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:32,239 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:32,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:32,239 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:32,239 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,339 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:32,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:32,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:32,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:32,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:32,343 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:32,343 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:32,343 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:32,343 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:32,343 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,343 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:32,345 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:32,345 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:32,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:32,345 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,345 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:32,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:32,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:32,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:32,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:32,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:32,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:32,346 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:32,346 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:32,346 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:32,346 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:32,346 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:32,346 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:32,346 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. started... 2023-04-27 03:54:32,347 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. as already flushing 2023-04-27 03:54:32,347 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:32,347 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:32,347 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:32,347 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,347 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:32,347 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:32,350 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,350 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:32,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:32,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:32,350 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:32,350 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:32,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:32,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:32,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:32,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:32,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:32,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@209b6b32[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:32,352 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:32,352 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:32,353 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:32,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:32,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:32,353 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:32,353 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:32,353 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,353 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:32,353 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:32,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:32,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:32,354 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:32,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:32,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:32,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:32,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:32,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:32,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:32,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:32,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:32,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:32,357 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:32,357 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:32,357 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:32,357 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:32,357 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:32,359 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,359 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:32,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:32,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,359 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:32,360 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:32,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:32,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:32,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:32,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:32,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:32,361 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:32,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:32,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,368 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:32,368 DEBUG [Thread-862] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x508246cc to 127.0.0.1:55020 2023-04-27 03:54:32,368 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,369 DEBUG [Thread-862] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:32,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:32,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:32,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:32,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:32,368 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:32,369 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:32,369 DEBUG [Thread-860] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00c2598b to 127.0.0.1:55020 2023-04-27 03:54:32,369 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:32,369 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:32,369 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:32,369 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:32,369 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:32,369 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:32,369 DEBUG [Thread-860] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:32,369 DEBUG [Thread-859] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:32,369 DEBUG [Thread-859] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:32,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:32,373 DEBUG [Thread-866] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x58e27a32 to 127.0.0.1:55020 2023-04-27 03:54:32,373 DEBUG [Thread-866] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:32,376 DEBUG [Thread-864] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08795451 to 127.0.0.1:55020 2023-04-27 03:54:32,376 DEBUG [Thread-864] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:32,436 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=838 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/cc9b78cb7bd044e18cdf9660b0a27284 2023-04-27 03:54:32,440 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/b3c0376b729641239275013377660973 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/b3c0376b729641239275013377660973 2023-04-27 03:54:32,443 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/b3c0376b729641239275013377660973, entries=150, sequenceid=838, filesize=25.4 K 2023-04-27 03:54:32,444 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/77bc02333f564c88956b99ed1f86c983 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/77bc02333f564c88956b99ed1f86c983 2023-04-27 03:54:32,447 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/77bc02333f564c88956b99ed1f86c983, entries=150, sequenceid=838, filesize=11.9 K 2023-04-27 03:54:32,448 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/cc9b78cb7bd044e18cdf9660b0a27284 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/cc9b78cb7bd044e18cdf9660b0a27284 2023-04-27 03:54:32,451 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/cc9b78cb7bd044e18cdf9660b0a27284, entries=150, sequenceid=838, filesize=11.9 K 2023-04-27 03:54:32,452 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for d66a88af1ba39e44423282d0b046d5b3 in 1657ms, sequenceid=838, compaction requested=true 2023-04-27 03:54:32,452 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:32,452 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:32,452 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:32,452 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:32,452 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:32,452 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:32,453 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78990 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:32,453 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:32,453 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): d66a88af1ba39e44423282d0b046d5b3/A is initiating minor compaction (all files) 2023-04-27 03:54:32,453 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): d66a88af1ba39e44423282d0b046d5b3/B is initiating minor compaction (all files) 2023-04-27 03:54:32,453 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d66a88af1ba39e44423282d0b046d5b3/A in TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:32,453 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d66a88af1ba39e44423282d0b046d5b3/B in TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:32,453 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/4138f95cbc5948b2ba5b1c76d2011538, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/0b46013496bf4d099a098ead819001d9, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/b3c0376b729641239275013377660973] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp, totalSize=77.1 K 2023-04-27 03:54:32,453 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/66ad8e143e4b424ea32e0f1c1d5ad4d2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/4024ac7bebfd4e9ba6fbd4b7d5d7a074, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/77bc02333f564c88956b99ed1f86c983] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp, totalSize=36.5 K 2023-04-27 03:54:32,453 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 4138f95cbc5948b2ba5b1c76d2011538, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=790, earliestPutTs=1682567668492 2023-04-27 03:54:32,453 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 66ad8e143e4b424ea32e0f1c1d5ad4d2, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=790, earliestPutTs=1682567668492 2023-04-27 03:54:32,454 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 0b46013496bf4d099a098ead819001d9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=827, earliestPutTs=1682567668553 2023-04-27 03:54:32,454 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 4024ac7bebfd4e9ba6fbd4b7d5d7a074, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=827, earliestPutTs=1682567668553 2023-04-27 03:54:32,454 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting b3c0376b729641239275013377660973, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=838, earliestPutTs=1682567670688 2023-04-27 03:54:32,454 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 77bc02333f564c88956b99ed1f86c983, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=838, earliestPutTs=1682567670688 2023-04-27 03:54:32,463 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): d66a88af1ba39e44423282d0b046d5b3#B#compaction#197 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:32,464 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): d66a88af1ba39e44423282d0b046d5b3#A#compaction#196 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:32,469 DEBUG [Thread-859] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:32,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:32,871 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/99da0726b29c412bb1224f9de1e3213e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/99da0726b29c412bb1224f9de1e3213e 2023-04-27 03:54:32,875 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d66a88af1ba39e44423282d0b046d5b3/B of d66a88af1ba39e44423282d0b046d5b3 into 99da0726b29c412bb1224f9de1e3213e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:32,875 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:32,875 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., storeName=d66a88af1ba39e44423282d0b046d5b3/B, priority=13, startTime=1682567672452; duration=0sec 2023-04-27 03:54:32,875 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:32,875 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:32,876 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:32,876 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): d66a88af1ba39e44423282d0b046d5b3/C is initiating minor compaction (all files) 2023-04-27 03:54:32,876 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d66a88af1ba39e44423282d0b046d5b3/C in TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:32,877 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/61b46a9a1b474d4bbde8a3f2ee2c35fa, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/8bd0a267ad58425190608d660acfb318, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/cc9b78cb7bd044e18cdf9660b0a27284] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp, totalSize=36.5 K 2023-04-27 03:54:32,877 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 61b46a9a1b474d4bbde8a3f2ee2c35fa, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=790, earliestPutTs=1682567668492 2023-04-27 03:54:32,877 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 8bd0a267ad58425190608d660acfb318, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=827, earliestPutTs=1682567668553 2023-04-27 03:54:32,878 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting cc9b78cb7bd044e18cdf9660b0a27284, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=838, earliestPutTs=1682567670688 2023-04-27 03:54:32,879 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/0b866817e5464fc58ae153a13242bb7f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/0b866817e5464fc58ae153a13242bb7f 2023-04-27 03:54:32,883 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d66a88af1ba39e44423282d0b046d5b3/A of d66a88af1ba39e44423282d0b046d5b3 into 0b866817e5464fc58ae153a13242bb7f(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:32,883 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): d66a88af1ba39e44423282d0b046d5b3#C#compaction#198 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:32,883 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:32,883 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., storeName=d66a88af1ba39e44423282d0b046d5b3/A, priority=13, startTime=1682567672452; duration=0sec 2023-04-27 03:54:32,883 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:32,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:32,952 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d66a88af1ba39e44423282d0b046d5b3 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-27 03:54:32,952 DEBUG [Thread-853] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21265fcb to 127.0.0.1:55020 2023-04-27 03:54:32,952 DEBUG [Thread-853] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:32,952 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=A 2023-04-27 03:54:32,952 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:32,952 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=B 2023-04-27 03:54:32,952 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:32,952 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=C 2023-04-27 03:54:32,952 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:32,954 DEBUG [Thread-851] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a71edf0 to 127.0.0.1:55020 2023-04-27 03:54:32,954 DEBUG [Thread-851] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:32,956 DEBUG [Thread-849] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x051c62b6 to 127.0.0.1:55020 2023-04-27 03:54:32,956 DEBUG [Thread-849] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:32,957 DEBUG [Thread-857] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0feb770d to 127.0.0.1:55020 2023-04-27 03:54:32,957 DEBUG [Thread-857] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:32,957 DEBUG [Thread-855] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7aba8e46 to 127.0.0.1:55020 2023-04-27 03:54:32,957 DEBUG [Thread-855] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:32,957 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-27 03:54:32,957 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 167 2023-04-27 03:54:32,957 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 148 2023-04-27 03:54:32,957 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 181 2023-04-27 03:54:32,958 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 134 2023-04-27 03:54:32,958 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 153 2023-04-27 03:54:32,958 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-27 03:54:32,958 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(398): read 4244 2023-04-27 03:54:32,958 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(398): read 4217 2023-04-27 03:54:32,958 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-27 03:54:32,958 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(402): scanned 1803 2023-04-27 03:54:32,958 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(403): verified 5409 rows 2023-04-27 03:54:32,958 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(402): scanned 1819 2023-04-27 03:54:32,958 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(403): verified 5457 rows 2023-04-27 03:54:32,958 INFO [Listener at localhost/33933] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-27 03:54:32,958 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5b8d0eb2 to 127.0.0.1:55020 2023-04-27 03:54:32,958 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:32,960 INFO [Listener at localhost/33933] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-27 03:54:32,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-27 03:54:32,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-27 03:54:32,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-27 03:54:32,965 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567672965"}]},"ts":"1682567672965"} 2023-04-27 03:54:32,966 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-27 03:54:33,063 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-27 03:54:33,065 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d66a88af1ba39e44423282d0b046d5b3, UNASSIGN}] 2023-04-27 03:54:33,065 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d66a88af1ba39e44423282d0b046d5b3, UNASSIGN 2023-04-27 03:54:33,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-27 03:54:33,066 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=d66a88af1ba39e44423282d0b046d5b3, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:33,066 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567673066"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682567673066"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682567673066"}]},"ts":"1682567673066"} 2023-04-27 03:54:33,067 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305}] 2023-04-27 03:54:33,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:33,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d66a88af1ba39e44423282d0b046d5b3, disabling compactions & flushes 2023-04-27 03:54:33,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 1 compactions & cache flush to complete for region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:33,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-27 03:54:33,291 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/fc845c49cfc04eb9a0d2b864ea20f74b as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/fc845c49cfc04eb9a0d2b864ea20f74b 2023-04-27 03:54:33,295 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d66a88af1ba39e44423282d0b046d5b3/C of d66a88af1ba39e44423282d0b046d5b3 into fc845c49cfc04eb9a0d2b864ea20f74b(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:33,295 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:33,295 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:33,295 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3., storeName=d66a88af1ba39e44423282d0b046d5b3/C, priority=13, startTime=1682567672452; duration=0sec 2023-04-27 03:54:33,295 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:33,373 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427300f2b91bf8e457c9feb091b86cd2e12 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427300f2b91bf8e457c9feb091b86cd2e12 2023-04-27 03:54:33,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-27 03:54:33,778 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=878, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/2b0b573188a94cffbb9a323a9e5ec941 2023-04-27 03:54:34,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-27 03:54:34,188 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=878 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/173b636a9abd483c80a379a828fc73e8 2023-04-27 03:54:34,598 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=878 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/f3ae86f6d2d04d21af83e3677d939fd5 2023-04-27 03:54:34,602 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/2b0b573188a94cffbb9a323a9e5ec941 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/2b0b573188a94cffbb9a323a9e5ec941 2023-04-27 03:54:34,606 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/2b0b573188a94cffbb9a323a9e5ec941, entries=150, sequenceid=878, filesize=25.4 K 2023-04-27 03:54:34,607 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/173b636a9abd483c80a379a828fc73e8 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/173b636a9abd483c80a379a828fc73e8 2023-04-27 03:54:34,611 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/173b636a9abd483c80a379a828fc73e8, entries=150, sequenceid=878, filesize=11.9 K 2023-04-27 03:54:34,611 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/f3ae86f6d2d04d21af83e3677d939fd5 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/f3ae86f6d2d04d21af83e3677d939fd5 2023-04-27 03:54:34,615 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/f3ae86f6d2d04d21af83e3677d939fd5, entries=150, sequenceid=878, filesize=11.9 K 2023-04-27 03:54:34,615 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=26.84 KB/27480 for d66a88af1ba39e44423282d0b046d5b3 in 1663ms, sequenceid=878, compaction requested=false 2023-04-27 03:54:34,616 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:34,616 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:34,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:34,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. after waiting 0 ms 2023-04-27 03:54:34,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:34,616 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing d66a88af1ba39e44423282d0b046d5b3 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-27 03:54:34,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=A 2023-04-27 03:54:34,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:34,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=B 2023-04-27 03:54:34,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:34,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d66a88af1ba39e44423282d0b046d5b3, store=C 2023-04-27 03:54:34,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:35,029 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427a1ea0b29df434edea83cbec9f45021fa to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427a1ea0b29df434edea83cbec9f45021fa 2023-04-27 03:54:35,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-27 03:54:35,434 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=886, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/eda26cdbfc26494c8e97af00def1bfca 2023-04-27 03:54:35,843 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=886 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/08d64d4e842d43ac884745bb30c99e0c 2023-04-27 03:54:36,253 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=886 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/da9607fc0f06434b95e400b9160d2cf3 2023-04-27 03:54:36,257 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/A/eda26cdbfc26494c8e97af00def1bfca as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/eda26cdbfc26494c8e97af00def1bfca 2023-04-27 03:54:36,260 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/eda26cdbfc26494c8e97af00def1bfca, entries=100, sequenceid=886, filesize=18.6 K 2023-04-27 03:54:36,261 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/B/08d64d4e842d43ac884745bb30c99e0c as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/08d64d4e842d43ac884745bb30c99e0c 2023-04-27 03:54:36,264 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/08d64d4e842d43ac884745bb30c99e0c, entries=100, sequenceid=886, filesize=9.5 K 2023-04-27 03:54:36,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/.tmp/C/da9607fc0f06434b95e400b9160d2cf3 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/da9607fc0f06434b95e400b9160d2cf3 2023-04-27 03:54:36,267 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/da9607fc0f06434b95e400b9160d2cf3, entries=100, sequenceid=886, filesize=9.5 K 2023-04-27 03:54:36,268 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for d66a88af1ba39e44423282d0b046d5b3 in 1652ms, sequenceid=886, compaction requested=true 2023-04-27 03:54:36,287 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/799b9c0b636e4d6da0d33ec2e2dda0cb, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/964d0f30c5a4487980ed4a60ee95adcb, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/b628d996fd6647b7bcabe35da1a1d79b, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/e6c0f4136fff4b7d99b76687617ef264, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/90beec4eb0bd4e52a1a794e3720913ff, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/1786be032664411dbd01f7729d7f63d2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/eacbc85e11f54c92ad93d692bf9246c8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/24e7bc7dd7504540913f4d5578b21e5c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/f7ec5aa9b3bd41b796fd6f744f8d4d96, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/dac46cdff3bf4472a323d30a772d237f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/6602ced3ae5340a4bcba65e865b26084, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/6844eb8c523c464db275b12d9bcecd3a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/76f1befaa3374353924e07230d307eee, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/a02ec880ad66426c952cc31f946d26b8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/ad74029298de45049031277a68acd07c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/5f0a6951ac814b769bd1ec7f172f7dbd, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/909907d368ac40faa7432ad75b201787, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/c6553da57498463eb5afaa4a67661b0d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/dfd120972de44a9eaca1cbbceb7f3654, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/951dd297ddd0440ab2d6d3dea0ee6b9a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/e6cf51cc68684827bdbf5eb1b1e4ef54, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/02b11277e5fd4278a11c6159d6234f74, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/16bebecbd3304e91a6b635f47188e041, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/74cb8aa4afb1460787e59a4f3be5f796, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/d19b3a7d39594026a2957e24f9b53b03, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/4138f95cbc5948b2ba5b1c76d2011538, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/0b46013496bf4d099a098ead819001d9, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/b3c0376b729641239275013377660973] to archive 2023-04-27 03:54:36,288 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 03:54:36,289 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/799b9c0b636e4d6da0d33ec2e2dda0cb to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/799b9c0b636e4d6da0d33ec2e2dda0cb 2023-04-27 03:54:36,290 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/964d0f30c5a4487980ed4a60ee95adcb to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/964d0f30c5a4487980ed4a60ee95adcb 2023-04-27 03:54:36,291 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/b628d996fd6647b7bcabe35da1a1d79b to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/b628d996fd6647b7bcabe35da1a1d79b 2023-04-27 03:54:36,293 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/e6c0f4136fff4b7d99b76687617ef264 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/e6c0f4136fff4b7d99b76687617ef264 2023-04-27 03:54:36,294 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/90beec4eb0bd4e52a1a794e3720913ff to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/90beec4eb0bd4e52a1a794e3720913ff 2023-04-27 03:54:36,296 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/1786be032664411dbd01f7729d7f63d2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/1786be032664411dbd01f7729d7f63d2 2023-04-27 03:54:36,297 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/eacbc85e11f54c92ad93d692bf9246c8 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/eacbc85e11f54c92ad93d692bf9246c8 2023-04-27 03:54:36,298 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/24e7bc7dd7504540913f4d5578b21e5c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/24e7bc7dd7504540913f4d5578b21e5c 2023-04-27 03:54:36,299 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/f7ec5aa9b3bd41b796fd6f744f8d4d96 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/f7ec5aa9b3bd41b796fd6f744f8d4d96 2023-04-27 03:54:36,300 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/dac46cdff3bf4472a323d30a772d237f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/dac46cdff3bf4472a323d30a772d237f 2023-04-27 03:54:36,301 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/6602ced3ae5340a4bcba65e865b26084 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/6602ced3ae5340a4bcba65e865b26084 2023-04-27 03:54:36,303 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/6844eb8c523c464db275b12d9bcecd3a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/6844eb8c523c464db275b12d9bcecd3a 2023-04-27 03:54:36,304 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/76f1befaa3374353924e07230d307eee to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/76f1befaa3374353924e07230d307eee 2023-04-27 03:54:36,305 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/a02ec880ad66426c952cc31f946d26b8 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/a02ec880ad66426c952cc31f946d26b8 2023-04-27 03:54:36,306 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/ad74029298de45049031277a68acd07c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/ad74029298de45049031277a68acd07c 2023-04-27 03:54:36,307 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/5f0a6951ac814b769bd1ec7f172f7dbd to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/5f0a6951ac814b769bd1ec7f172f7dbd 2023-04-27 03:54:36,308 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/909907d368ac40faa7432ad75b201787 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/909907d368ac40faa7432ad75b201787 2023-04-27 03:54:36,309 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/c6553da57498463eb5afaa4a67661b0d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/c6553da57498463eb5afaa4a67661b0d 2023-04-27 03:54:36,310 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/dfd120972de44a9eaca1cbbceb7f3654 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/dfd120972de44a9eaca1cbbceb7f3654 2023-04-27 03:54:36,311 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/951dd297ddd0440ab2d6d3dea0ee6b9a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/951dd297ddd0440ab2d6d3dea0ee6b9a 2023-04-27 03:54:36,312 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/e6cf51cc68684827bdbf5eb1b1e4ef54 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/e6cf51cc68684827bdbf5eb1b1e4ef54 2023-04-27 03:54:36,313 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/02b11277e5fd4278a11c6159d6234f74 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/02b11277e5fd4278a11c6159d6234f74 2023-04-27 03:54:36,314 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/16bebecbd3304e91a6b635f47188e041 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/16bebecbd3304e91a6b635f47188e041 2023-04-27 03:54:36,315 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/74cb8aa4afb1460787e59a4f3be5f796 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/74cb8aa4afb1460787e59a4f3be5f796 2023-04-27 03:54:36,316 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/d19b3a7d39594026a2957e24f9b53b03 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/d19b3a7d39594026a2957e24f9b53b03 2023-04-27 03:54:36,317 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/4138f95cbc5948b2ba5b1c76d2011538 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/4138f95cbc5948b2ba5b1c76d2011538 2023-04-27 03:54:36,318 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/0b46013496bf4d099a098ead819001d9 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/0b46013496bf4d099a098ead819001d9 2023-04-27 03:54:36,319 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/b3c0376b729641239275013377660973 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/b3c0376b729641239275013377660973 2023-04-27 03:54:36,322 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/f437fc8d4fd04f9a931f64fbbf76e4aa, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/060b290ce2444baf903cd48fc3f55a69, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/97d8cc4120f44b16981de814a6ca81f5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/42b8796756d44e55a37c1f9e67d28071, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/991219af4c9b46388020cece1ca5fbe9, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/428beccef25046b6b484be6b0d325449, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/88753208d5d74c6eba0522e2948b3a99, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/48656692b4df43469050053de8e079c6, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/cf41efa3cad64b1980e035e7a02056aa, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/6cdd4a851aa048d683cbb9b38d2e9ea2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/b4ac997bec3d4296938ffbb05da5bb4c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/3629b732a6554761a9460f960db3cd2a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/ca39d6000e1f48509d96e37057d90dcf, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/be553bcc8b0d4de9b0955978b3273762, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/d356c57c6d81449f98be5bdd4efb71e3, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/4479c57b5ff542ca904782887d659452, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/56f953307da24777861c852e8c4459e2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/1abece4aabde483b94257778e96dd68a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/850210a3fee34082ba33a108919fc096, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/62d575e93c6f4b76b8c9ff1e8d1a2d53, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/83116b4a1eaa4231bab25fef8f75081f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/e31b8bc0f16f4c10b498eb798f26f37a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/c37fbec62e7d4702bdba9317dec2d990, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/8b083bbd20e54bc4878f0bfd78f46633, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/66ad8e143e4b424ea32e0f1c1d5ad4d2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/4c1512a9fba34205bf52da4cccc6b400, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/4024ac7bebfd4e9ba6fbd4b7d5d7a074, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/77bc02333f564c88956b99ed1f86c983] to archive 2023-04-27 03:54:36,323 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 03:54:36,324 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/f437fc8d4fd04f9a931f64fbbf76e4aa to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/f437fc8d4fd04f9a931f64fbbf76e4aa 2023-04-27 03:54:36,325 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/060b290ce2444baf903cd48fc3f55a69 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/060b290ce2444baf903cd48fc3f55a69 2023-04-27 03:54:36,326 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/97d8cc4120f44b16981de814a6ca81f5 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/97d8cc4120f44b16981de814a6ca81f5 2023-04-27 03:54:36,327 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/42b8796756d44e55a37c1f9e67d28071 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/42b8796756d44e55a37c1f9e67d28071 2023-04-27 03:54:36,328 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/991219af4c9b46388020cece1ca5fbe9 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/991219af4c9b46388020cece1ca5fbe9 2023-04-27 03:54:36,329 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/428beccef25046b6b484be6b0d325449 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/428beccef25046b6b484be6b0d325449 2023-04-27 03:54:36,330 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/88753208d5d74c6eba0522e2948b3a99 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/88753208d5d74c6eba0522e2948b3a99 2023-04-27 03:54:36,331 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/48656692b4df43469050053de8e079c6 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/48656692b4df43469050053de8e079c6 2023-04-27 03:54:36,332 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/cf41efa3cad64b1980e035e7a02056aa to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/cf41efa3cad64b1980e035e7a02056aa 2023-04-27 03:54:36,333 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/6cdd4a851aa048d683cbb9b38d2e9ea2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/6cdd4a851aa048d683cbb9b38d2e9ea2 2023-04-27 03:54:36,334 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/b4ac997bec3d4296938ffbb05da5bb4c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/b4ac997bec3d4296938ffbb05da5bb4c 2023-04-27 03:54:36,335 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/3629b732a6554761a9460f960db3cd2a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/3629b732a6554761a9460f960db3cd2a 2023-04-27 03:54:36,335 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/ca39d6000e1f48509d96e37057d90dcf to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/ca39d6000e1f48509d96e37057d90dcf 2023-04-27 03:54:36,336 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/be553bcc8b0d4de9b0955978b3273762 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/be553bcc8b0d4de9b0955978b3273762 2023-04-27 03:54:36,337 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/d356c57c6d81449f98be5bdd4efb71e3 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/d356c57c6d81449f98be5bdd4efb71e3 2023-04-27 03:54:36,338 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/4479c57b5ff542ca904782887d659452 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/4479c57b5ff542ca904782887d659452 2023-04-27 03:54:36,339 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/56f953307da24777861c852e8c4459e2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/56f953307da24777861c852e8c4459e2 2023-04-27 03:54:36,340 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/1abece4aabde483b94257778e96dd68a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/1abece4aabde483b94257778e96dd68a 2023-04-27 03:54:36,341 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/850210a3fee34082ba33a108919fc096 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/850210a3fee34082ba33a108919fc096 2023-04-27 03:54:36,341 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/62d575e93c6f4b76b8c9ff1e8d1a2d53 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/62d575e93c6f4b76b8c9ff1e8d1a2d53 2023-04-27 03:54:36,342 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/83116b4a1eaa4231bab25fef8f75081f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/83116b4a1eaa4231bab25fef8f75081f 2023-04-27 03:54:36,343 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/e31b8bc0f16f4c10b498eb798f26f37a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/e31b8bc0f16f4c10b498eb798f26f37a 2023-04-27 03:54:36,344 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/c37fbec62e7d4702bdba9317dec2d990 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/c37fbec62e7d4702bdba9317dec2d990 2023-04-27 03:54:36,345 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/8b083bbd20e54bc4878f0bfd78f46633 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/8b083bbd20e54bc4878f0bfd78f46633 2023-04-27 03:54:36,346 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/66ad8e143e4b424ea32e0f1c1d5ad4d2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/66ad8e143e4b424ea32e0f1c1d5ad4d2 2023-04-27 03:54:36,347 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/4c1512a9fba34205bf52da4cccc6b400 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/4c1512a9fba34205bf52da4cccc6b400 2023-04-27 03:54:36,348 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/4024ac7bebfd4e9ba6fbd4b7d5d7a074 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/4024ac7bebfd4e9ba6fbd4b7d5d7a074 2023-04-27 03:54:36,349 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/77bc02333f564c88956b99ed1f86c983 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/77bc02333f564c88956b99ed1f86c983 2023-04-27 03:54:36,351 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/c5fd75961933470990d4b94c6ebf7c13, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/a745bc211a4442e4a8136783cf4b4ed6, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/81630c9d75e14440889a470c0b95bddd, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/b7ce1d66c36c4fb7988a3031334cc076, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/06dc6270b31e498fad233379f8fb6091, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/5e16a187dc8b4957ae3aab370f24be74, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/332f94de8bae488d8825be338483139a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/62ed9585f56d49489011bb981df1cfb8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/fd82c02d0cc541b99055083636d38488, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/0f7229f465bf43a5ba2a1bcc3c887829, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/f164d30330584b3783fc2ae838d0fbe2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/e82ded12040f4f668a30eaa63e137fe8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/91986859bcfc46e68ffccaaeec4c6b60, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/52961e2ca07c4ce9adcd71a9d01283e5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/71d5448509884ab588753bc75c4ae4d0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/251d34b133d34d6b923d7c40c0dbd153, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/e057e8fdc33e4598aec10b77f5debd72, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/ac17cc391be240559c55732aa70ab185, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/4503675ee1974f838f7ac0418a203e33, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/122250d39b384ab29709f768e0f1dfa0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/5e7f958a13454b1e810f6657f4b7419a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/6d6f68d70be44fa3bfdc1d3afb68c7ab, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/1cf0bbd249164846abb885641ba00fe9, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/61b46a9a1b474d4bbde8a3f2ee2c35fa, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/6a1d840517fe4bde831ee8cde17059fa, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/8bd0a267ad58425190608d660acfb318, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/cc9b78cb7bd044e18cdf9660b0a27284] to archive 2023-04-27 03:54:36,352 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 03:54:36,353 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/c5fd75961933470990d4b94c6ebf7c13 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/c5fd75961933470990d4b94c6ebf7c13 2023-04-27 03:54:36,354 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/a745bc211a4442e4a8136783cf4b4ed6 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/a745bc211a4442e4a8136783cf4b4ed6 2023-04-27 03:54:36,356 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/81630c9d75e14440889a470c0b95bddd to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/81630c9d75e14440889a470c0b95bddd 2023-04-27 03:54:36,357 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/b7ce1d66c36c4fb7988a3031334cc076 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/b7ce1d66c36c4fb7988a3031334cc076 2023-04-27 03:54:36,358 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/06dc6270b31e498fad233379f8fb6091 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/06dc6270b31e498fad233379f8fb6091 2023-04-27 03:54:36,358 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/5e16a187dc8b4957ae3aab370f24be74 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/5e16a187dc8b4957ae3aab370f24be74 2023-04-27 03:54:36,359 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/332f94de8bae488d8825be338483139a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/332f94de8bae488d8825be338483139a 2023-04-27 03:54:36,360 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/62ed9585f56d49489011bb981df1cfb8 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/62ed9585f56d49489011bb981df1cfb8 2023-04-27 03:54:36,361 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/fd82c02d0cc541b99055083636d38488 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/fd82c02d0cc541b99055083636d38488 2023-04-27 03:54:36,362 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/0f7229f465bf43a5ba2a1bcc3c887829 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/0f7229f465bf43a5ba2a1bcc3c887829 2023-04-27 03:54:36,363 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/f164d30330584b3783fc2ae838d0fbe2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/f164d30330584b3783fc2ae838d0fbe2 2023-04-27 03:54:36,364 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/e82ded12040f4f668a30eaa63e137fe8 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/e82ded12040f4f668a30eaa63e137fe8 2023-04-27 03:54:36,365 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/91986859bcfc46e68ffccaaeec4c6b60 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/91986859bcfc46e68ffccaaeec4c6b60 2023-04-27 03:54:36,366 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/52961e2ca07c4ce9adcd71a9d01283e5 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/52961e2ca07c4ce9adcd71a9d01283e5 2023-04-27 03:54:36,367 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/71d5448509884ab588753bc75c4ae4d0 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/71d5448509884ab588753bc75c4ae4d0 2023-04-27 03:54:36,368 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/251d34b133d34d6b923d7c40c0dbd153 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/251d34b133d34d6b923d7c40c0dbd153 2023-04-27 03:54:36,369 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/e057e8fdc33e4598aec10b77f5debd72 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/e057e8fdc33e4598aec10b77f5debd72 2023-04-27 03:54:36,370 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/ac17cc391be240559c55732aa70ab185 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/ac17cc391be240559c55732aa70ab185 2023-04-27 03:54:36,371 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/4503675ee1974f838f7ac0418a203e33 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/4503675ee1974f838f7ac0418a203e33 2023-04-27 03:54:36,372 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/122250d39b384ab29709f768e0f1dfa0 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/122250d39b384ab29709f768e0f1dfa0 2023-04-27 03:54:36,373 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/5e7f958a13454b1e810f6657f4b7419a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/5e7f958a13454b1e810f6657f4b7419a 2023-04-27 03:54:36,374 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/6d6f68d70be44fa3bfdc1d3afb68c7ab to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/6d6f68d70be44fa3bfdc1d3afb68c7ab 2023-04-27 03:54:36,375 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/1cf0bbd249164846abb885641ba00fe9 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/1cf0bbd249164846abb885641ba00fe9 2023-04-27 03:54:36,376 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/61b46a9a1b474d4bbde8a3f2ee2c35fa to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/61b46a9a1b474d4bbde8a3f2ee2c35fa 2023-04-27 03:54:36,377 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/6a1d840517fe4bde831ee8cde17059fa to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/6a1d840517fe4bde831ee8cde17059fa 2023-04-27 03:54:36,378 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/8bd0a267ad58425190608d660acfb318 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/8bd0a267ad58425190608d660acfb318 2023-04-27 03:54:36,379 DEBUG [StoreCloser-TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/cc9b78cb7bd044e18cdf9660b0a27284 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/cc9b78cb7bd044e18cdf9660b0a27284 2023-04-27 03:54:36,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/recovered.edits/889.seqid, newMaxSeqId=889, maxSeqId=4 2023-04-27 03:54:36,384 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3. 2023-04-27 03:54:36,384 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d66a88af1ba39e44423282d0b046d5b3: 2023-04-27 03:54:36,386 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:36,386 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=d66a88af1ba39e44423282d0b046d5b3, regionState=CLOSED 2023-04-27 03:54:36,386 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567676386"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682567676386"}]},"ts":"1682567676386"} 2023-04-27 03:54:36,388 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-27 03:54:36,388 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure d66a88af1ba39e44423282d0b046d5b3, server=jenkins-hbase4.apache.org,40885,1682567618305 in 3.3200 sec 2023-04-27 03:54:36,389 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-27 03:54:36,389 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d66a88af1ba39e44423282d0b046d5b3, UNASSIGN in 3.3240 sec 2023-04-27 03:54:36,390 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567676390"}]},"ts":"1682567676390"} 2023-04-27 03:54:36,391 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-27 03:54:36,392 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-27 03:54:36,393 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 3.4320 sec 2023-04-27 03:54:37,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-27 03:54:37,071 INFO [Listener at localhost/33933] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-27 03:54:37,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-27 03:54:37,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:54:37,074 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:54:37,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-27 03:54:37,075 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:54:37,079 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:37,082 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A, FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B, FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C, FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/recovered.edits] 2023-04-27 03:54:37,084 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/0b866817e5464fc58ae153a13242bb7f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/0b866817e5464fc58ae153a13242bb7f 2023-04-27 03:54:37,085 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/2b0b573188a94cffbb9a323a9e5ec941 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/2b0b573188a94cffbb9a323a9e5ec941 2023-04-27 03:54:37,086 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/eda26cdbfc26494c8e97af00def1bfca to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/A/eda26cdbfc26494c8e97af00def1bfca 2023-04-27 03:54:37,089 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/08d64d4e842d43ac884745bb30c99e0c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/08d64d4e842d43ac884745bb30c99e0c 2023-04-27 03:54:37,090 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/173b636a9abd483c80a379a828fc73e8 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/173b636a9abd483c80a379a828fc73e8 2023-04-27 03:54:37,091 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/99da0726b29c412bb1224f9de1e3213e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/B/99da0726b29c412bb1224f9de1e3213e 2023-04-27 03:54:37,093 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/da9607fc0f06434b95e400b9160d2cf3 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/da9607fc0f06434b95e400b9160d2cf3 2023-04-27 03:54:37,094 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/f3ae86f6d2d04d21af83e3677d939fd5 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/f3ae86f6d2d04d21af83e3677d939fd5 2023-04-27 03:54:37,095 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/fc845c49cfc04eb9a0d2b864ea20f74b to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/C/fc845c49cfc04eb9a0d2b864ea20f74b 2023-04-27 03:54:37,097 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/recovered.edits/889.seqid to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3/recovered.edits/889.seqid 2023-04-27 03:54:37,098 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/d66a88af1ba39e44423282d0b046d5b3 2023-04-27 03:54:37,098 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-27 03:54:37,098 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-27 03:54:37,099 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-27 03:54:37,102 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042710ae3bc1a8b84da6ba1b61a23f4c1d68 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042710ae3bc1a8b84da6ba1b61a23f4c1d68 2023-04-27 03:54:37,103 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272244e81f8cf34dcfab2652a523313ee2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272244e81f8cf34dcfab2652a523313ee2 2023-04-27 03:54:37,103 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272ae4f172315b487d9633135b4def1c44 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272ae4f172315b487d9633135b4def1c44 2023-04-27 03:54:37,104 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272e4101d935ac4bbf9dddf7f69742d4c7 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272e4101d935ac4bbf9dddf7f69742d4c7 2023-04-27 03:54:37,105 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272e68d86f82d442deaa0c7acf41ee3c2d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272e68d86f82d442deaa0c7acf41ee3c2d 2023-04-27 03:54:37,106 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427300f2b91bf8e457c9feb091b86cd2e12 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427300f2b91bf8e457c9feb091b86cd2e12 2023-04-27 03:54:37,107 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304273019652855b44597bbff8fbd7de9791d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304273019652855b44597bbff8fbd7de9791d 2023-04-27 03:54:37,108 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042738c6ee75434142eaa18a3ed08875f2d2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042738c6ee75434142eaa18a3ed08875f2d2 2023-04-27 03:54:37,109 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304274735a865f21e4c3894289d854983beed to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304274735a865f21e4c3894289d854983beed 2023-04-27 03:54:37,110 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427499e79e45d5444668200597f744ddbff to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427499e79e45d5444668200597f744ddbff 2023-04-27 03:54:37,111 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427537675995a4f4c9caef3a8880a628e5f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427537675995a4f4c9caef3a8880a628e5f 2023-04-27 03:54:37,112 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042776f1ac940c374cd783fa4ff54ef04b7d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042776f1ac940c374cd783fa4ff54ef04b7d 2023-04-27 03:54:37,113 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427823685dabc924758bc1096956e298c9a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427823685dabc924758bc1096956e298c9a 2023-04-27 03:54:37,114 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427876af8d6a40a40d18db36ff6d094e904 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427876af8d6a40a40d18db36ff6d094e904 2023-04-27 03:54:37,115 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427878846bc885545debb1f5843845fdcec to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427878846bc885545debb1f5843845fdcec 2023-04-27 03:54:37,116 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042788dfa8e4b9204dca8d70094235bc4b0c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042788dfa8e4b9204dca8d70094235bc4b0c 2023-04-27 03:54:37,117 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304279615bb7d05c346ed8207100d63ad4165 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304279615bb7d05c346ed8207100d63ad4165 2023-04-27 03:54:37,118 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304279b66090538b64aa894eef2407d2b9d4c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304279b66090538b64aa894eef2407d2b9d4c 2023-04-27 03:54:37,119 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427a1ea0b29df434edea83cbec9f45021fa to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427a1ea0b29df434edea83cbec9f45021fa 2023-04-27 03:54:37,120 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427d81bee79475249888090a1d00c7ae067 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427d81bee79475249888090a1d00c7ae067 2023-04-27 03:54:37,121 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427e3807a7cad5f4464bad06deb128e9ea1 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427e3807a7cad5f4464bad06deb128e9ea1 2023-04-27 03:54:37,122 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427f3fb1a725abd4f91be12bba2b10cb9a0 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427f3fb1a725abd4f91be12bba2b10cb9a0 2023-04-27 03:54:37,123 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427fd4064b1204741b0abd30d0034f92c53 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427fd4064b1204741b0abd30d0034f92c53 2023-04-27 03:54:37,124 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-27 03:54:37,125 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:54:37,127 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-27 03:54:37,129 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-27 03:54:37,130 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:54:37,130 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-27 03:54:37,130 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682567677130"}]},"ts":"9223372036854775807"} 2023-04-27 03:54:37,132 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-27 03:54:37,132 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => d66a88af1ba39e44423282d0b046d5b3, NAME => 'TestAcidGuarantees,,1682567650198.d66a88af1ba39e44423282d0b046d5b3.', STARTKEY => '', ENDKEY => ''}] 2023-04-27 03:54:37,132 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-27 03:54:37,132 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682567677132"}]},"ts":"9223372036854775807"} 2023-04-27 03:54:37,133 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-27 03:54:37,135 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:54:37,136 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 63 msec 2023-04-27 03:54:37,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-27 03:54:37,176 INFO [Listener at localhost/33933] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-27 03:54:37,188 INFO [Listener at localhost/33933] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=235 (was 235), OpenFileDescriptor=578 (was 569) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=366 (was 421), ProcessCount=170 (was 170), AvailableMemoryMB=11989 (was 12216) 2023-04-27 03:54:37,199 INFO [Listener at localhost/33933] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=235, OpenFileDescriptor=578, MaxFileDescriptor=60000, SystemLoadAverage=366, ProcessCount=170, AvailableMemoryMB=11989 2023-04-27 03:54:37,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-27 03:54:37,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-27 03:54:37,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-27 03:54:37,204 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-27 03:54:37,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-27 03:54:37,204 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-27 03:54:37,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-27 03:54:37,205 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:37,206 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7 empty. 2023-04-27 03:54:37,206 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:37,206 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-27 03:54:37,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-27 03:54:37,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-27 03:54:37,614 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-27 03:54:37,616 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => e07ddca27cd19d46379b6939f22f78c7, NAME => 'TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp 2023-04-27 03:54:37,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-27 03:54:38,022 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 03:54:38,022 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing e07ddca27cd19d46379b6939f22f78c7, disabling compactions & flushes 2023-04-27 03:54:38,022 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:38,022 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:38,022 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. after waiting 0 ms 2023-04-27 03:54:38,022 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:38,022 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:38,022 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:38,024 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-27 03:54:38,025 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567678025"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682567678025"}]},"ts":"1682567678025"} 2023-04-27 03:54:38,026 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-27 03:54:38,027 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-27 03:54:38,027 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567678027"}]},"ts":"1682567678027"} 2023-04-27 03:54:38,027 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-27 03:54:38,030 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e07ddca27cd19d46379b6939f22f78c7, ASSIGN}] 2023-04-27 03:54:38,031 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e07ddca27cd19d46379b6939f22f78c7, ASSIGN 2023-04-27 03:54:38,032 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e07ddca27cd19d46379b6939f22f78c7, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40885,1682567618305; forceNewPlan=false, retain=false 2023-04-27 03:54:38,183 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=e07ddca27cd19d46379b6939f22f78c7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:38,184 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567678183"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682567678183"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682567678183"}]},"ts":"1682567678183"} 2023-04-27 03:54:38,185 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305}] 2023-04-27 03:54:38,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-27 03:54:38,340 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:38,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e07ddca27cd19d46379b6939f22f78c7, NAME => 'TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.', STARTKEY => '', ENDKEY => ''} 2023-04-27 03:54:38,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:38,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 03:54:38,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:38,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:38,341 INFO [StoreOpener-e07ddca27cd19d46379b6939f22f78c7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:38,342 DEBUG [StoreOpener-e07ddca27cd19d46379b6939f22f78c7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A 2023-04-27 03:54:38,343 DEBUG [StoreOpener-e07ddca27cd19d46379b6939f22f78c7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A 2023-04-27 03:54:38,343 INFO [StoreOpener-e07ddca27cd19d46379b6939f22f78c7-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:54:38,343 INFO [StoreOpener-e07ddca27cd19d46379b6939f22f78c7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e07ddca27cd19d46379b6939f22f78c7 columnFamilyName A 2023-04-27 03:54:38,344 INFO [StoreOpener-e07ddca27cd19d46379b6939f22f78c7-1] regionserver.HStore(310): Store=e07ddca27cd19d46379b6939f22f78c7/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:54:38,344 INFO [StoreOpener-e07ddca27cd19d46379b6939f22f78c7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:38,345 DEBUG [StoreOpener-e07ddca27cd19d46379b6939f22f78c7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B 2023-04-27 03:54:38,345 DEBUG [StoreOpener-e07ddca27cd19d46379b6939f22f78c7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B 2023-04-27 03:54:38,345 INFO [StoreOpener-e07ddca27cd19d46379b6939f22f78c7-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:54:38,345 INFO [StoreOpener-e07ddca27cd19d46379b6939f22f78c7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e07ddca27cd19d46379b6939f22f78c7 columnFamilyName B 2023-04-27 03:54:38,345 INFO [StoreOpener-e07ddca27cd19d46379b6939f22f78c7-1] regionserver.HStore(310): Store=e07ddca27cd19d46379b6939f22f78c7/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:54:38,345 INFO [StoreOpener-e07ddca27cd19d46379b6939f22f78c7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:38,346 DEBUG [StoreOpener-e07ddca27cd19d46379b6939f22f78c7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C 2023-04-27 03:54:38,346 DEBUG [StoreOpener-e07ddca27cd19d46379b6939f22f78c7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C 2023-04-27 03:54:38,347 INFO [StoreOpener-e07ddca27cd19d46379b6939f22f78c7-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:54:38,347 INFO [StoreOpener-e07ddca27cd19d46379b6939f22f78c7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e07ddca27cd19d46379b6939f22f78c7 columnFamilyName C 2023-04-27 03:54:38,347 INFO [StoreOpener-e07ddca27cd19d46379b6939f22f78c7-1] regionserver.HStore(310): Store=e07ddca27cd19d46379b6939f22f78c7/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:54:38,347 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:38,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:38,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:38,350 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-27 03:54:38,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:38,352 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-27 03:54:38,352 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e07ddca27cd19d46379b6939f22f78c7; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61656881, jitterRate=-0.08124087750911713}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-27 03:54:38,352 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:38,353 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., pid=30, masterSystemTime=1682567678336 2023-04-27 03:54:38,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:38,354 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:38,355 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=e07ddca27cd19d46379b6939f22f78c7, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:38,355 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567678354"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682567678354"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682567678354"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682567678354"}]},"ts":"1682567678354"} 2023-04-27 03:54:38,357 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-27 03:54:38,357 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 in 170 msec 2023-04-27 03:54:38,358 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-27 03:54:38,358 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e07ddca27cd19d46379b6939f22f78c7, ASSIGN in 327 msec 2023-04-27 03:54:38,359 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-27 03:54:38,359 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567678359"}]},"ts":"1682567678359"} 2023-04-27 03:54:38,359 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-27 03:54:38,361 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-27 03:54:38,363 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1610 sec 2023-04-27 03:54:39,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-27 03:54:39,309 INFO [Listener at localhost/33933] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-27 03:54:39,310 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x262f4570 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:54:39,315 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23c97092, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:54:39,316 DEBUG [Listener at localhost/33933] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:54:39,318 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37782, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:54:39,319 DEBUG [Listener at localhost/33933] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-27 03:54:39,320 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48870, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-27 03:54:39,321 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x00ca898c to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:54:39,325 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c124820, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:54:39,325 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x6585a70a to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:54:39,328 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3157d4df, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:54:39,329 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x7480f9da to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:54:39,334 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3cf5bdb7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:54:39,334 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x1a24d617 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:54:39,338 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6919b2a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:54:39,338 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x104dbf4e to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:54:39,342 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ce5573e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:54:39,342 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x087b3095 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:54:39,346 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ea1ca03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:54:39,346 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x12115052 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:54:39,350 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@433f251d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:54:39,351 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x7e65578e to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:54:39,357 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66ee0bd0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:54:39,357 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x6fcbaba8 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:54:39,361 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b4658f8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:54:39,361 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x6f8ff99f to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:54:39,366 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d2b992d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:54:39,370 DEBUG [hconnection-0x1cbd6042-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:54:39,370 DEBUG [hconnection-0x624c369d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:54:39,371 DEBUG [hconnection-0xfff38f2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:54:39,371 DEBUG [hconnection-0x5aba5340-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:54:39,372 DEBUG [hconnection-0x3be52075-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:54:39,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:39,372 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37798, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:54:39,372 DEBUG [hconnection-0x506cd27c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:54:39,373 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37830, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:54:39,376 DEBUG [hconnection-0x76e2503d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:54:39,377 DEBUG [hconnection-0x5a76c1b1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:54:39,376 DEBUG [hconnection-0x7f5857e3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:54:39,377 DEBUG [hconnection-0x39be9191-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:54:39,381 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37796, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:54:39,381 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37788, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:54:39,381 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37824, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:54:39,381 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37864, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:54:39,382 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37812, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:54:39,383 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37856, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:54:39,383 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37840, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:54:39,395 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37876, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:54:39,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:39,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:39,403 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:39,404 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:39,404 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:39,404 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:39,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:39,406 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e07ddca27cd19d46379b6939f22f78c7 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-27 03:54:39,408 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,408 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,409 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=A 2023-04-27 03:54:39,409 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:39,409 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=B 2023-04-27 03:54:39,409 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:39,409 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=C 2023-04-27 03:54:39,409 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:39,410 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:39,410 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,411 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,411 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:39,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:39,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:39,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:39,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:39,422 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:39,422 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:39,422 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:39,428 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:39,428 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:39,431 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:39,431 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:39,431 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:39,431 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:39,431 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:39,431 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:39,431 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:39,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567739429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,433 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9554, totalHeapSize=21976, min timestamp=1682567679426, max timestamp=1682567679431Num uniques 859; 2023-04-27 03:54:39,433 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=9554, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:54:39,433 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9554, totalHeapSize=21976, min timestamp=1682567679426, max timestamp=1682567679431Num uniques 859; 2023-04-27 03:54:39,433 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=183, cellSize=8368, totalHeapSize=19272, min timestamp=1682567679426, max timestamp=1682567679431Num uniques 859; 2023-04-27 03:54:39,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=8368, suffix heap size=117064,new segment heap size=19272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=183 2023-04-27 03:54:39,433 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=9554, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:54:39,440 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:39,440 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:39,442 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,442 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:39,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:39,442 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,442 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:39,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:39,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:39,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:39,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@753073ab[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:39,445 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:39,445 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,446 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,446 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:39,447 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:39,447 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:39,447 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,447 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:39,453 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:39,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567739450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:39,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567739452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,454 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:39,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:39,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:39,454 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:39,455 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:39,455 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:39,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:39,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:39,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:39,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37876 deadline: 1682567739453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:39,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:39,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:39,457 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:39,457 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:39,457 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:39,457 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:39,457 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:39,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567739454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,459 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,459 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,459 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:39,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:39,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:39,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:39,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:39,460 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:39,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:39,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,461 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,464 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:39,464 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:39,464 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:39,464 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:39,464 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:39,465 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,465 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,465 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,465 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,465 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:39,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:39,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:39,465 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:39,465 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:39,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:39,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:39,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:39,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567739536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:39,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567739555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:39,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567739555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:39,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37876 deadline: 1682567739557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:39,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567739558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,565 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:39,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:39,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:39,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:39,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:39,569 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:39,569 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:39,569 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:39,569 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:39,569 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,569 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,570 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,570 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:39,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:39,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:39,571 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,571 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:39,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:39,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,572 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:39,572 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:39,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:39,572 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:39,572 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:39,572 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:39,573 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:39,573 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:39,573 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:39,573 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:39,573 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:39,573 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:39,573 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:39,573 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:39,573 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,573 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:39,573 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:39,576 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,576 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:39,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:39,576 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,576 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:39,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:39,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:39,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:39,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:39,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@329b8c37[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:39,577 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:39,577 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,579 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,579 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:39,579 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:39,579 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:39,579 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,579 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:39,581 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:39,581 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:39,581 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:39,581 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:39,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:39,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:39,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:39,583 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:39,584 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:39,584 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:39,584 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:39,584 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:39,585 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,585 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,585 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:39,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:39,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:39,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:39,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:39,586 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:39,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:39,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,587 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:39,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,590 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:39,591 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:39,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:39,591 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:39,591 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,591 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,591 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,591 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,591 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,591 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,591 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:39,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:39,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:39,591 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:39,591 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:39,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:39,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:39,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:39,691 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:39,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:39,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:39,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:39,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:39,695 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:39,695 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:39,695 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:39,695 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:39,695 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,695 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,697 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:39,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:39,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:39,697 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,697 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,697 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:39,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:39,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,698 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:39,698 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:39,698 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:39,698 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:39,699 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:39,699 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:39,699 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:39,699 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:39,699 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:39,699 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:39,699 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:39,699 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:39,699 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:39,699 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:39,699 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,699 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:39,699 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:39,701 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:39,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:39,702 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,702 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,702 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:39,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:39,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:39,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:39,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:39,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@311d46a0[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:39,703 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:39,704 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,705 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,705 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:39,705 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,705 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:39,705 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:39,705 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:39,707 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,707 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:39,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:39,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:39,707 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:39,707 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:39,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:39,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:39,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:39,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:39,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:39,709 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:39,709 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:39,709 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:39,709 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:39,710 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:39,712 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,712 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:39,712 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:39,712 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:39,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:39,713 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:39,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,715 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,715 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:39,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:39,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:39,716 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,716 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:39,716 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:39,716 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:39,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:39,716 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:39,716 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:39,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:39,716 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,716 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,716 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,716 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:39,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:39,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:39,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567739739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:39,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567739758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:39,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567739759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:39,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567739761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:39,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37876 deadline: 1682567739761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,816 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:39,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:39,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:39,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:39,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:39,819 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:39,819 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:39,819 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:39,820 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:39,820 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,820 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,821 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:39,821 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:39,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:39,821 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,821 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:39,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:39,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:39,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:39,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:39,822 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:39,822 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:39,822 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:39,823 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:39,823 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:39,823 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:39,823 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:39,823 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:39,823 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:39,823 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:39,823 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:39,823 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,823 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:39,823 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:39,825 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,825 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:39,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:39,826 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,826 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:39,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:39,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:39,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:39,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:39,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4aa4794a[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:39,827 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:39,827 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,828 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,828 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:39,828 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:39,828 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,829 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:39,828 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:39,833 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:39,833 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,833 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:39,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:39,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:39,834 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:39,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:39,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:39,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:39,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:39,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:39,836 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:39,836 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:39,836 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:39,836 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:39,836 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:39,837 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,838 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:39,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:39,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,838 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:39,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:39,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:39,838 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:39,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:39,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,839 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:39,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,843 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,844 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:39,844 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:39,844 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:39,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:39,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:39,844 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:39,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:39,844 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,844 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:39,844 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:39,844 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,844 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,844 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,844 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:39,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:39,849 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/1defe950142546b7acdc141d69d1f2a0 2023-04-27 03:54:39,944 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:39,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:39,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:39,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:39,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:39,947 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:39,947 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:39,947 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:39,947 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:39,948 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,948 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,950 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:39,950 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:39,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:39,950 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,950 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:39,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:39,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:39,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:39,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:39,951 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:39,951 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:39,951 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:39,951 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:39,951 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:39,951 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:39,951 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:39,951 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:39,951 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:39,952 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:39,952 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:39,952 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,952 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:39,952 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:39,953 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,953 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:39,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:39,953 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,953 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:39,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:39,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:39,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:39,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:39,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cfff69e[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:39,955 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:39,955 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,956 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,956 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:39,956 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:39,956 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:39,956 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,956 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:39,957 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:39,957 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:39,957 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,957 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:39,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:39,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:39,960 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:39,960 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:39,960 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:39,960 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:39,960 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:39,962 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,962 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,962 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:39,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:39,963 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:39,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,964 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:39,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:39,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,966 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:39,966 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:39,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:39,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:39,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:39,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:39,966 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,966 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:39,967 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,967 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:39,967 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:39,967 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,967 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:39,967 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:39,967 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:39,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:39,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:39,967 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:39,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:40,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:40,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567740042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:40,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567740062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:40,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567740062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:40,067 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:40,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567740065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:40,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37876 deadline: 1682567740065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:40,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:40,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:40,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:40,070 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:40,070 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:40,070 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:40,070 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:40,071 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,071 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:40,072 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:40,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:40,072 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:40,072 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,072 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:40,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:40,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:40,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:40,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:40,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:40,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:40,074 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:40,074 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:40,074 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:40,074 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:40,074 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:40,074 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:40,074 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:40,074 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:40,074 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:40,074 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:40,074 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:40,074 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,074 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:40,074 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:40,076 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,076 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:40,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:40,076 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,076 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:40,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:40,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:40,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:40,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:40,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:40,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@526b2f02[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:40,078 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:40,078 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,081 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,081 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:40,081 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:40,081 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:40,081 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,081 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:40,082 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:40,082 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:40,082 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,082 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:40,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:40,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:40,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:40,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:40,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:40,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:40,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:40,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:40,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:40,085 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:40,085 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:40,085 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:40,085 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:40,085 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:40,087 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,087 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:40,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:40,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,087 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:40,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:40,087 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:40,089 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:40,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:40,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:40,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,091 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:40,091 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:40,091 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:40,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:40,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:40,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:40,091 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:40,091 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:40,091 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:40,091 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:40,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:40,091 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:40,091 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:40,092 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:40,092 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,092 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,192 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:40,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:40,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:40,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:40,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:40,195 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:40,195 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:40,195 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:40,195 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:40,195 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,195 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:40,197 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,197 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:40,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:40,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:40,197 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,197 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:40,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:40,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:40,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:40,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:40,199 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:40,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:40,199 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:40,199 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:40,199 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:40,199 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:40,199 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:40,199 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:40,199 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:40,199 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:40,199 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:40,199 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:40,199 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:40,199 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,199 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:40,199 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:40,201 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,201 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:40,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:40,201 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,201 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:40,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:40,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:40,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:40,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:40,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:40,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9985ba9[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:40,203 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:40,203 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,204 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,204 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:40,204 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:40,204 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:40,204 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,204 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:40,205 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:40,205 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,205 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:40,206 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:40,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:40,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:40,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:40,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:40,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:40,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:40,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:40,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:40,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:40,208 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:40,208 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:40,208 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:40,208 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:40,208 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:40,211 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,211 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:40,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:40,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,211 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:40,211 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:40,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:40,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:40,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:40,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:40,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:40,212 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:40,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:40,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,217 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:40,217 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:40,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:40,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:40,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:40,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:40,217 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:40,217 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:40,217 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:40,218 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:40,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:40,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:40,218 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:40,218 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:40,218 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,218 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,218 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,218 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,262 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/4dedea72d0764e3ea1b91fd8cf436c4e 2023-04-27 03:54:40,293 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/a697c54514184d41baf50ee9f1271b13 2023-04-27 03:54:40,297 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/1defe950142546b7acdc141d69d1f2a0 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/1defe950142546b7acdc141d69d1f2a0 2023-04-27 03:54:40,302 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/1defe950142546b7acdc141d69d1f2a0, entries=150, sequenceid=13, filesize=11.6 K 2023-04-27 03:54:40,303 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/4dedea72d0764e3ea1b91fd8cf436c4e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/4dedea72d0764e3ea1b91fd8cf436c4e 2023-04-27 03:54:40,307 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/4dedea72d0764e3ea1b91fd8cf436c4e, entries=150, sequenceid=13, filesize=11.6 K 2023-04-27 03:54:40,308 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/a697c54514184d41baf50ee9f1271b13 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/a697c54514184d41baf50ee9f1271b13 2023-04-27 03:54:40,311 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/a697c54514184d41baf50ee9f1271b13, entries=150, sequenceid=13, filesize=11.6 K 2023-04-27 03:54:40,312 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=119.60 KB/122474 for e07ddca27cd19d46379b6939f22f78c7 in 906ms, sequenceid=13, compaction requested=false 2023-04-27 03:54:40,312 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:40,318 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:40,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:40,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:40,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:40,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:40,321 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:40,321 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:40,321 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:40,321 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:40,322 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,322 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:40,323 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,323 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:40,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:40,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:40,324 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,324 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:40,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:40,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:40,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:40,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:40,325 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:40,325 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:40,325 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:40,325 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:40,325 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:40,328 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:40,328 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:40,328 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:40,328 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:40,328 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e07ddca27cd19d46379b6939f22f78c7 3/3 column families, dataSize=119.60 KB heapSize=337.81 KB 2023-04-27 03:54:40,328 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=A 2023-04-27 03:54:40,328 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:40,328 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=B 2023-04-27 03:54:40,328 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:40,329 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=C 2023-04-27 03:54:40,329 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:40,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:40,550 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:40,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:40,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567740567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:40,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567740568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:40,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567740569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:40,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567740569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:40,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37876 deadline: 1682567740569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,672 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:40,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567740672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:40,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567740673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:40,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567740673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,736 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/736cc4ad1c4e46ce88c61b01eba7af34 2023-04-27 03:54:40,749 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/488315f1704f430cb5bc47fc04526c90 2023-04-27 03:54:40,759 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=39.10 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/50ec9bc5727c42d3b0a713c28adf09b9 2023-04-27 03:54:40,763 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/736cc4ad1c4e46ce88c61b01eba7af34 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/736cc4ad1c4e46ce88c61b01eba7af34 2023-04-27 03:54:40,767 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/736cc4ad1c4e46ce88c61b01eba7af34, entries=150, sequenceid=47, filesize=11.6 K 2023-04-27 03:54:40,767 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/488315f1704f430cb5bc47fc04526c90 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/488315f1704f430cb5bc47fc04526c90 2023-04-27 03:54:40,770 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/488315f1704f430cb5bc47fc04526c90, entries=150, sequenceid=47, filesize=11.6 K 2023-04-27 03:54:40,771 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/50ec9bc5727c42d3b0a713c28adf09b9 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/50ec9bc5727c42d3b0a713c28adf09b9 2023-04-27 03:54:40,774 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/50ec9bc5727c42d3b0a713c28adf09b9, entries=150, sequenceid=47, filesize=11.6 K 2023-04-27 03:54:40,775 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~119.60 KB/122474, heapSize ~337.77 KB/345872, currentSize=60.38 KB/61830 for e07ddca27cd19d46379b6939f22f78c7 in 447ms, sequenceid=47, compaction requested=false 2023-04-27 03:54:40,775 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:40,775 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:40,775 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:40,775 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:40,775 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,775 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:40,776 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:40,778 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,778 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:40,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:40,779 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:40,779 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:40,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:40,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:40,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:40,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:40,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@609f1df9[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:40,780 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:40,780 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,781 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,782 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:40,782 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:40,782 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:40,782 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,782 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:40,783 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:40,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:40,783 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:40,784 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:40,784 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:40,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:40,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:40,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:40,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:40,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:40,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:40,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:40,786 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:40,786 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:40,786 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:40,786 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:40,786 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:40,789 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,789 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,789 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:40,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:40,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:40,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:40,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:40,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:40,790 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:40,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:40,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:40,791 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:40,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:40,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,793 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:40,793 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:40,793 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:40,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:40,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:40,793 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:40,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:40,793 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:40,793 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,793 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,793 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,793 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:40,793 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:40,793 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:40,794 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:40,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:40,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:40,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:40,878 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e07ddca27cd19d46379b6939f22f78c7 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-27 03:54:40,878 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=A 2023-04-27 03:54:40,878 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:40,878 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=B 2023-04-27 03:54:40,878 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:40,878 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=C 2023-04-27 03:54:40,878 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:40,887 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/249d0f4e024b48d4acdd7613bad9d86c 2023-04-27 03:54:40,894 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:40,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:40,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:40,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:40,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:40,897 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:40,898 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:40,898 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:40,898 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:40,898 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,898 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:40,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:40,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567740895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,899 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:40,899 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:40,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:40,900 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,900 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:40,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:40,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:40,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:40,900 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:40,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:40,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:40,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:40,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567740897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,901 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:40,901 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:40,901 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:40,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:40,901 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:40,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567740898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,901 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:40,901 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:40,901 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:40,901 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:40,901 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:40,901 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:40,901 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:40,901 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,901 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:40,901 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:40,903 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,903 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:40,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:40,903 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,903 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:40,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:40,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:40,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:40,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:40,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:40,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23975c72[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:40,904 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:40,904 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,907 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,907 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:40,907 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,907 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:40,907 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:40,907 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:40,909 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:40,909 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:40,909 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,909 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:40,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:40,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:40,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:40,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:40,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:40,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:40,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:40,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:40,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:40,911 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:40,911 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:40,911 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:40,911 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:40,911 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:40,912 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,913 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:40,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,913 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:40,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:40,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:40,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:40,913 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:40,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:40,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:40,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:40,914 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:40,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:40,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,917 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:40,917 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,917 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:40,917 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:40,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:40,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:40,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:40,917 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:40,917 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:40,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:40,917 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:40,917 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:40,917 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:40,917 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,917 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:40,917 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:40,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:40,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:41,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:41,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567740999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:41,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567741002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:41,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567741002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,017 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:41,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:41,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:41,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:41,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:41,021 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:41,021 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:41,021 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:41,021 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:41,021 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,021 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,023 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:41,023 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:41,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:41,023 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,023 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:41,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:41,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:41,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:41,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:41,024 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:41,024 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:41,024 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:41,024 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:41,024 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:41,024 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:41,024 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:41,025 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:41,025 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:41,025 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:41,025 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:41,025 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,025 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:41,025 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:41,026 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,026 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:41,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:41,027 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,027 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:41,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:41,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:41,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:41,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:41,028 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:41,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@280f743d[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:41,028 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,029 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,030 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:41,030 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:41,030 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:41,030 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,030 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:41,032 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,032 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:41,032 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:41,032 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:41,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:41,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:41,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:41,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:41,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:41,035 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:41,035 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:41,035 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:41,035 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:41,035 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:41,036 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,036 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,036 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:41,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:41,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:41,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:41,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:41,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,037 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:41,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:41,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,040 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:41,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,046 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,046 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,046 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:41,046 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:41,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:41,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:41,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:41,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:41,046 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:41,046 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,046 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:41,046 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:41,046 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,046 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,046 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,046 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:41,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:41,146 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:41,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:41,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:41,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:41,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:41,150 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:41,151 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:41,151 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:41,151 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:41,151 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,151 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,153 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,153 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:41,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:41,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:41,154 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,154 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:41,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:41,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:41,155 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:41,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:41,155 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:41,155 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:41,155 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:41,155 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:41,155 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:41,155 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:41,155 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:41,155 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:41,155 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:41,155 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:41,155 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:41,155 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,155 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:41,155 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:41,157 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:41,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:41,157 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:41,158 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,158 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:41,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:41,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:41,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:41,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63813786[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:41,159 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:41,159 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,160 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,160 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:41,160 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:41,160 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,160 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:41,160 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:41,161 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,161 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:41,161 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:41,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:41,161 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:41,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:41,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:41,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:41,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:41,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:41,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:41,164 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:41,164 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:41,164 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:41,164 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:41,164 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:41,166 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,166 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:41,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:41,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,166 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:41,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:41,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:41,166 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:41,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:41,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,168 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:41,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,170 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:41,170 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:41,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:41,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:41,170 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,171 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,171 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:41,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:41,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:41,171 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,171 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,171 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,171 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,171 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:41,171 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:41,171 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:41,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:41,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567741203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:41,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567741204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:41,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567741205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,271 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:41,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:41,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:41,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:41,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:41,274 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:41,275 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:41,275 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:41,275 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:41,275 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,275 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,277 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,277 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:41,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:41,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:41,277 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,277 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:41,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:41,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:41,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:41,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:41,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:41,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:41,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:41,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:41,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:41,278 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:41,279 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:41,279 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:41,279 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:41,279 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:41,279 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:41,279 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,279 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:41,279 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:41,281 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,281 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:41,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:41,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:41,281 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,281 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:41,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:41,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:41,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:41,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b699eda[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:41,282 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:41,282 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,284 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,284 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:41,284 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:41,285 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,285 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:41,285 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:41,286 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,286 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:41,286 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:41,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,286 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:41,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:41,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:41,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:41,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:41,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:41,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:41,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:41,289 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:41,289 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:41,289 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:41,289 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:41,289 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:41,290 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,291 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,291 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:41,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:41,291 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:41,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,292 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:41,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,295 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,295 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:41,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:41,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:41,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:41,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:41,295 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:41,295 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,295 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,295 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,295 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,295 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,295 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:41,295 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:41,295 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:41,295 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:41,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:41,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:41,297 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/46ced774acbb4d5c8c6d81e1f35fdcb0 2023-04-27 03:54:41,308 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/53782d67c7cf47ef8e0adde64d6d129e 2023-04-27 03:54:41,312 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/249d0f4e024b48d4acdd7613bad9d86c as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/249d0f4e024b48d4acdd7613bad9d86c 2023-04-27 03:54:41,315 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/249d0f4e024b48d4acdd7613bad9d86c, entries=200, sequenceid=61, filesize=13.9 K 2023-04-27 03:54:41,316 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/46ced774acbb4d5c8c6d81e1f35fdcb0 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/46ced774acbb4d5c8c6d81e1f35fdcb0 2023-04-27 03:54:41,321 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/46ced774acbb4d5c8c6d81e1f35fdcb0, entries=150, sequenceid=61, filesize=11.6 K 2023-04-27 03:54:41,322 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/53782d67c7cf47ef8e0adde64d6d129e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/53782d67c7cf47ef8e0adde64d6d129e 2023-04-27 03:54:41,326 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/53782d67c7cf47ef8e0adde64d6d129e, entries=150, sequenceid=61, filesize=11.6 K 2023-04-27 03:54:41,327 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for e07ddca27cd19d46379b6939f22f78c7 in 450ms, sequenceid=61, compaction requested=true 2023-04-27 03:54:41,327 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:41,327 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:41,327 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:41,327 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:41,327 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:41,327 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:41,328 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:41,328 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/A is initiating minor compaction (all files) 2023-04-27 03:54:41,328 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/A in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:41,328 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:41,329 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/B is initiating minor compaction (all files) 2023-04-27 03:54:41,329 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/B in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:41,328 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/1defe950142546b7acdc141d69d1f2a0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/736cc4ad1c4e46ce88c61b01eba7af34, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/249d0f4e024b48d4acdd7613bad9d86c] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=37.1 K 2023-04-27 03:54:41,329 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/4dedea72d0764e3ea1b91fd8cf436c4e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/488315f1704f430cb5bc47fc04526c90, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/46ced774acbb4d5c8c6d81e1f35fdcb0] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=34.8 K 2023-04-27 03:54:41,329 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 1defe950142546b7acdc141d69d1f2a0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682567679395 2023-04-27 03:54:41,329 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 4dedea72d0764e3ea1b91fd8cf436c4e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682567679395 2023-04-27 03:54:41,330 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 736cc4ad1c4e46ce88c61b01eba7af34, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1682567679445 2023-04-27 03:54:41,330 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 488315f1704f430cb5bc47fc04526c90, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1682567679445 2023-04-27 03:54:41,330 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 249d0f4e024b48d4acdd7613bad9d86c, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1682567680560 2023-04-27 03:54:41,330 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 46ced774acbb4d5c8c6d81e1f35fdcb0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1682567680560 2023-04-27 03:54:41,342 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#A#compaction#214 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:41,353 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#B#compaction#215 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:41,396 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:41,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:41,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:41,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:41,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:41,400 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:41,400 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:41,400 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:41,400 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:41,401 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,401 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,403 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:41,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:41,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:41,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,403 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,403 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,404 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:41,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:41,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:41,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:41,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:41,411 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:41,411 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:41,411 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:41,411 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:41,412 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:41,412 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:41,412 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:41,412 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e07ddca27cd19d46379b6939f22f78c7 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-27 03:54:41,412 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=A 2023-04-27 03:54:41,412 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:41,412 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=B 2023-04-27 03:54:41,412 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:41,412 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=C 2023-04-27 03:54:41,412 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:41,421 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/1c56857fd3334b388a6e075e21a2a624 2023-04-27 03:54:41,437 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/fe4909d3acce4e1ba695fd3975f508bb 2023-04-27 03:54:41,450 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/ac74401ef24748218ee7b68344137a75 2023-04-27 03:54:41,455 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/1c56857fd3334b388a6e075e21a2a624 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/1c56857fd3334b388a6e075e21a2a624 2023-04-27 03:54:41,459 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/1c56857fd3334b388a6e075e21a2a624, entries=150, sequenceid=79, filesize=11.6 K 2023-04-27 03:54:41,460 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/fe4909d3acce4e1ba695fd3975f508bb as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/fe4909d3acce4e1ba695fd3975f508bb 2023-04-27 03:54:41,463 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/fe4909d3acce4e1ba695fd3975f508bb, entries=150, sequenceid=79, filesize=11.6 K 2023-04-27 03:54:41,464 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/ac74401ef24748218ee7b68344137a75 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/ac74401ef24748218ee7b68344137a75 2023-04-27 03:54:41,468 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/ac74401ef24748218ee7b68344137a75, entries=150, sequenceid=79, filesize=11.6 K 2023-04-27 03:54:41,468 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for e07ddca27cd19d46379b6939f22f78c7 in 56ms, sequenceid=79, compaction requested=true 2023-04-27 03:54:41,469 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:41,469 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:41,469 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:41,469 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:41,469 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:41,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:41,473 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,474 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,474 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,474 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:41,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:41,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:41,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:41,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:41,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:41,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:41,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62843909[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:41,476 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:41,476 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,477 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:41,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,477 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:41,478 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:41,478 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:41,479 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,479 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:41,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:41,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:41,479 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:41,479 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:41,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:41,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:41,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:41,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:41,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:41,482 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:41,482 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:41,482 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:41,482 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:41,482 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:41,485 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,485 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:41,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:41,485 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:41,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:41,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:41,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,486 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:41,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:41,488 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:41,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,490 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,490 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,490 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:41,490 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:41,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:41,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:41,490 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:41,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:41,490 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:41,490 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:41,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:41,490 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:41,490 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,491 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:41,491 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,491 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:41,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:41,513 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e07ddca27cd19d46379b6939f22f78c7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:54:41,513 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=A 2023-04-27 03:54:41,513 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:41,513 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=B 2023-04-27 03:54:41,513 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:41,514 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=C 2023-04-27 03:54:41,514 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:41,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9558, totalHeapSize=21976, min timestamp=1682567681531, max timestamp=1682567681540Num uniques 859; 2023-04-27 03:54:41,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9554, totalHeapSize=21976, min timestamp=1682567681531, max timestamp=1682567681540Num uniques 859; 2023-04-27 03:54:41,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39328, new segment data size=9558, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:54:41,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=9554, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:54:41,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1682567681531, max timestamp=1682567681540Num uniques 859; 2023-04-27 03:54:41,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39327, new segment data size=7185, suffix heap size=117064,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=157 2023-04-27 03:54:41,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:41,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567741579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:41,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37876 deadline: 1682567741580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:41,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567741580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:41,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567741581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:41,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567741581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,591 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:41,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:41,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:41,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:41,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:41,594 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:41,594 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:41,594 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:41,594 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:41,594 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,594 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,598 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:41,598 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:41,598 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,598 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:41,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:41,599 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:41,599 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:41,599 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:41,599 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:41,599 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:41,599 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:41,599 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:41,599 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:41,599 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:41,599 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:41,599 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:41,599 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:41,600 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:41,600 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,600 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:41,600 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:41,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,602 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:41,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:41,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:41,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:41,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:41,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:41,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:41,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31f907f8[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:41,604 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:41,604 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,605 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,605 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:41,605 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:41,605 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,605 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:41,605 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:41,606 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:41,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:41,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:41,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:41,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:41,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:41,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:41,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:41,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:41,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:41,608 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:41,608 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:41,608 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:41,608 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:41,608 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:41,610 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,610 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,610 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:41,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:41,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:41,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:41,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:41,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:41,611 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:41,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,612 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:41,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,615 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:41,615 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:41,615 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,615 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:41,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:41,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:41,615 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:41,615 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:41,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,616 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:41,616 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:41,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:41,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:41,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:41,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567741684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:41,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567741685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:41,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567741687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:41,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567741687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,716 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:41,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:41,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:41,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:41,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:41,719 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:41,719 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:41,719 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:41,719 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:41,720 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,720 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,721 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:41,721 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:41,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:41,721 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,722 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:41,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:41,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:41,723 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:41,723 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:41,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:41,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:41,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:41,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:41,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:41,723 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:41,723 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:41,723 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:41,723 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:41,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:41,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:41,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:41,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:41,725 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,725 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:41,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:41,725 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,725 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:41,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:41,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:41,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:41,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:41,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bae2b46[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:41,726 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:41,726 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,727 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,728 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:41,727 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:41,728 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:41,728 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:41,730 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:41,730 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:41,730 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,730 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:41,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:41,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:41,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:41,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:41,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:41,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:41,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:41,732 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:41,732 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:41,732 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:41,733 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:41,733 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:41,734 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,734 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:41,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:41,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,734 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:41,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:41,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:41,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:41,735 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:41,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:41,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,738 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:41,738 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:41,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:41,738 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,738 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:41,738 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:41,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:41,738 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:41,738 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:41,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:41,739 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,739 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:41,739 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,739 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:41,739 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,766 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/a892a37d297f473bb5feeac38963586f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/a892a37d297f473bb5feeac38963586f 2023-04-27 03:54:41,774 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/f4b63fb7cdca454791d6eb3159bc844d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/f4b63fb7cdca454791d6eb3159bc844d 2023-04-27 03:54:41,774 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/A of e07ddca27cd19d46379b6939f22f78c7 into a892a37d297f473bb5feeac38963586f(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:41,774 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:41,774 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/A, priority=13, startTime=1682567681327; duration=0sec 2023-04-27 03:54:41,774 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:41,775 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 03:54:41,776 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 03:54:41,776 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/C is initiating minor compaction (all files) 2023-04-27 03:54:41,776 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/C in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:41,776 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/a697c54514184d41baf50ee9f1271b13, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/50ec9bc5727c42d3b0a713c28adf09b9, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/53782d67c7cf47ef8e0adde64d6d129e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/ac74401ef24748218ee7b68344137a75] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=46.4 K 2023-04-27 03:54:41,776 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting a697c54514184d41baf50ee9f1271b13, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682567679395 2023-04-27 03:54:41,777 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 50ec9bc5727c42d3b0a713c28adf09b9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1682567679445 2023-04-27 03:54:41,777 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 53782d67c7cf47ef8e0adde64d6d129e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1682567680560 2023-04-27 03:54:41,778 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting ac74401ef24748218ee7b68344137a75, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1682567680894 2023-04-27 03:54:41,779 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/B of e07ddca27cd19d46379b6939f22f78c7 into f4b63fb7cdca454791d6eb3159bc844d(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:41,779 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:41,779 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/B, priority=13, startTime=1682567681327; duration=0sec 2023-04-27 03:54:41,779 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:41,785 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#C#compaction#220 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:41,839 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:41,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:41,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:41,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:41,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:41,842 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:41,843 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:41,843 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:41,843 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:41,843 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,843 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,845 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,845 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:41,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:41,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:41,845 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,845 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:41,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:41,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,846 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:41,846 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:41,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:41,846 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:41,846 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:41,846 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:41,847 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:41,847 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:41,847 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:41,847 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:41,847 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:41,847 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:41,847 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:41,847 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:41,847 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,847 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:41,847 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:41,850 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,850 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:41,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:41,850 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,850 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:41,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:41,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:41,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:41,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:41,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c86e1a7[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:41,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:41,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,853 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,853 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:41,853 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:41,853 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:41,854 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,854 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:41,855 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:41,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:41,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:41,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:41,857 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:41,857 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:41,857 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:41,857 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:41,857 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:41,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:41,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:41,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:41,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:41,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:41,860 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:41,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,861 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:41,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:41,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:41,864 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:41,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:41,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:41,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:41,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:41,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,865 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:41,865 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:41,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:41,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:41,865 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,866 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,866 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,866 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:41,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567741886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:41,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567741888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:41,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567741889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:41,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567741889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,912 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-27 03:54:41,922 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=89 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/4ee12e19ebb34b4ab45b7e932ff97f26 2023-04-27 03:54:41,965 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:41,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:41,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:41,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:41,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:41,968 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:41,968 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:41,968 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:41,969 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:41,969 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,969 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,970 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,970 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:41,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:41,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:41,971 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,971 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:41,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:41,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,972 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:41,972 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:41,972 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:41,972 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:41,972 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:41,972 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:41,972 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:41,972 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:41,972 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:41,972 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:41,972 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:41,972 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:41,973 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:41,973 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:41,973 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,973 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:41,973 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:41,974 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,975 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:41,975 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,975 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:41,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:41,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:41,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:41,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f89d14f[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:41,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,983 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:41,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:41,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:41,985 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:41,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:41,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:41,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:41,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:41,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:41,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:41,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:41,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:41,987 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:41,988 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:41,988 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:41,988 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:41,988 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:41,989 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,989 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:41,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:41,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,989 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:41,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:41,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:41,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:41,990 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:41,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,992 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:41,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:41,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,994 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,994 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,994 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:41,994 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:41,994 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:41,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:41,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:41,994 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:41,994 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,994 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:41,994 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:41,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:41,995 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:41,994 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:41,995 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:41,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:41,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:41,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:42,095 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:42,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:42,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:42,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:42,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:42,098 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:42,098 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:42,098 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:42,098 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:42,099 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,099 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:42,106 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:42,106 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:42,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:42,106 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,106 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:42,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:42,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:42,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:42,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:42,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:42,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:42,107 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:42,107 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:42,107 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:42,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:42,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:42,108 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:42,108 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:42,108 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:42,108 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:42,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:42,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:42,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:42,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:42,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,110 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:42,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:42,110 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,110 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:42,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:42,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:42,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:42,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:42,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:42,111 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:42,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5636d581[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:42,111 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,113 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,113 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:42,113 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:42,113 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:42,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:42,114 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:42,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:42,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:42,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:42,115 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:42,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:42,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:42,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:42,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:42,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:42,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:42,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:42,117 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:42,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:42,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:42,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:42,117 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:42,119 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,119 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,119 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:42,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:42,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:42,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:42,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:42,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:42,120 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:42,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:42,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:42,121 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:42,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:42,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,123 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,123 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:42,123 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:42,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:42,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:42,123 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:42,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:42,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:42,123 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:42,123 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:42,123 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,123 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,123 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:42,123 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:42,123 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:42,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:42,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:42,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:42,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567742189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,194 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:42,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567742192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:42,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567742193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:42,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567742193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,198 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/e4630e195f424a30a4b8d6298320d6f5 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/e4630e195f424a30a4b8d6298320d6f5 2023-04-27 03:54:42,202 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/C of e07ddca27cd19d46379b6939f22f78c7 into e4630e195f424a30a4b8d6298320d6f5(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:42,202 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:42,202 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/C, priority=12, startTime=1682567681327; duration=0sec 2023-04-27 03:54:42,203 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:42,224 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:42,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:42,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:42,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:42,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:42,227 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:42,227 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:42,227 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:42,227 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:42,227 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,228 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:42,229 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,229 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:42,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:42,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:42,229 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,229 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:42,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:42,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:42,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:42,230 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:42,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:42,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:42,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:42,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:42,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:42,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:42,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:42,231 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:42,231 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:42,231 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:42,231 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:42,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:42,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:42,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:42,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:42,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:42,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:42,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:42,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:42,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:42,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:42,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:42,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:42,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c68301a[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:42,235 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:42,235 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,237 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,237 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:42,237 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:42,237 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,237 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:42,237 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:42,238 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:42,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:42,239 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:42,239 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:42,239 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:42,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:42,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:42,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:42,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:42,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:42,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:42,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:42,241 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:42,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:42,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:42,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:42,241 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:42,242 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,242 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:42,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,243 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:42,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:42,243 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:42,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:42,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:42,244 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:42,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:42,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,246 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,246 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:42,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:42,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,247 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:42,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:42,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:42,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:42,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:42,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:42,247 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:42,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:42,247 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:42,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:42,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:42,345 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=89 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/503fd21105694441a9ca1e0d978a29ea 2023-04-27 03:54:42,347 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:42,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:42,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:42,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:42,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:42,354 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:42,354 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:42,354 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:42,354 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:42,355 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,355 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:42,356 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=89 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/27a6125fd7d9416596dc8e22fb587295 2023-04-27 03:54:42,356 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:42,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:42,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:42,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:42,357 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,357 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:42,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:42,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:42,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:42,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:42,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:42,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:42,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:42,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:42,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:42,359 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:42,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:42,359 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:42,359 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:42,359 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:42,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:42,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:42,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:42,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:42,360 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/4ee12e19ebb34b4ab45b7e932ff97f26 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/4ee12e19ebb34b4ab45b7e932ff97f26 2023-04-27 03:54:42,361 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:42,361 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:42,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:42,362 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,362 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:42,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:42,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:42,364 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/4ee12e19ebb34b4ab45b7e932ff97f26, entries=150, sequenceid=89, filesize=11.6 K 2023-04-27 03:54:42,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:42,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:42,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27b87de9[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:42,364 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:42,365 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,365 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/503fd21105694441a9ca1e0d978a29ea as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/503fd21105694441a9ca1e0d978a29ea 2023-04-27 03:54:42,366 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,366 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:42,366 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:42,366 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,366 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:42,366 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:42,369 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/503fd21105694441a9ca1e0d978a29ea, entries=150, sequenceid=89, filesize=11.6 K 2023-04-27 03:54:42,369 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/27a6125fd7d9416596dc8e22fb587295 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/27a6125fd7d9416596dc8e22fb587295 2023-04-27 03:54:42,371 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:42,371 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:42,371 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,371 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:42,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:42,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:42,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:42,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:42,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:42,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:42,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:42,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:42,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:42,374 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:42,374 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:42,374 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:42,374 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/27a6125fd7d9416596dc8e22fb587295, entries=150, sequenceid=89, filesize=11.6 K 2023-04-27 03:54:42,374 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:42,374 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:42,375 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.86 KB/135028 for e07ddca27cd19d46379b6939f22f78c7 in 862ms, sequenceid=89, compaction requested=true 2023-04-27 03:54:42,375 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:42,375 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:42,376 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:42,376 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:42,376 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:42,376 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:42,377 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:42,377 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/A is initiating minor compaction (all files) 2023-04-27 03:54:42,377 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/A in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:42,377 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/a892a37d297f473bb5feeac38963586f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/1c56857fd3334b388a6e075e21a2a624, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/4ee12e19ebb34b4ab45b7e932ff97f26] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=34.9 K 2023-04-27 03:54:42,377 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:42,377 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/B is initiating minor compaction (all files) 2023-04-27 03:54:42,377 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/B in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:42,377 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/f4b63fb7cdca454791d6eb3159bc844d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/fe4909d3acce4e1ba695fd3975f508bb, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/503fd21105694441a9ca1e0d978a29ea] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=34.9 K 2023-04-27 03:54:42,378 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting f4b63fb7cdca454791d6eb3159bc844d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1682567680560 2023-04-27 03:54:42,378 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting a892a37d297f473bb5feeac38963586f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1682567680560 2023-04-27 03:54:42,378 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting fe4909d3acce4e1ba695fd3975f508bb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1682567680894 2023-04-27 03:54:42,379 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 1c56857fd3334b388a6e075e21a2a624, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1682567680894 2023-04-27 03:54:42,379 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 503fd21105694441a9ca1e0d978a29ea, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1682567681511 2023-04-27 03:54:42,379 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 4ee12e19ebb34b4ab45b7e932ff97f26, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1682567681511 2023-04-27 03:54:42,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:42,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:42,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:42,381 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:42,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:42,384 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,384 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:42,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:42,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:42,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:42,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:42,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:42,386 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,388 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:42,388 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:42,388 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:42,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:42,388 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:42,388 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#B#compaction#223 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:42,388 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:42,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,388 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:42,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:42,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:42,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,389 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:42,389 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:42,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:42,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:42,391 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#A#compaction#224 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:42,489 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:42,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:42,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:42,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:42,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:42,492 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:42,492 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:42,492 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:42,492 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:42,493 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,493 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:42,494 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:42,494 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:42,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:42,495 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,495 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:42,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:42,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:42,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:42,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:42,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:42,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:42,496 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:42,496 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:42,496 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:42,496 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:42,496 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:42,497 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:42,497 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:42,497 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e07ddca27cd19d46379b6939f22f78c7 3/3 column families, dataSize=131.86 KB heapSize=375.02 KB 2023-04-27 03:54:42,497 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=A 2023-04-27 03:54:42,497 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:42,497 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=B 2023-04-27 03:54:42,497 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:42,497 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=C 2023-04-27 03:54:42,497 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:42,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:42,698 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:42,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:42,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567742703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:42,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567742706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:42,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567742706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:42,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567742707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,798 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/144e4278319d42b88af5042aab61a92f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/144e4278319d42b88af5042aab61a92f 2023-04-27 03:54:42,802 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/B of e07ddca27cd19d46379b6939f22f78c7 into 144e4278319d42b88af5042aab61a92f(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:42,802 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:42,802 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/B, priority=13, startTime=1682567682375; duration=0sec 2023-04-27 03:54:42,803 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:42,803 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-27 03:54:42,804 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-27 03:54:42,804 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-27 03:54:42,804 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. because compaction request was cancelled 2023-04-27 03:54:42,806 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/bb9da04d8b684408b943cdcb7e5c77cd as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/bb9da04d8b684408b943cdcb7e5c77cd 2023-04-27 03:54:42,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:42,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567742807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:42,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567742810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:42,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:42,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567742810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567742810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,812 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/A of e07ddca27cd19d46379b6939f22f78c7 into bb9da04d8b684408b943cdcb7e5c77cd(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:42,812 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:42,812 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/A, priority=13, startTime=1682567682375; duration=0sec 2023-04-27 03:54:42,813 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:42,905 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/93722a2a9be5466cb3425a7ad982983f 2023-04-27 03:54:42,915 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/4e49b5ea8cc142e3a0c83fd3c6931c28 2023-04-27 03:54:42,947 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.41 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/6cb3ee473c4144e580f8d546b01700bf 2023-04-27 03:54:42,953 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/93722a2a9be5466cb3425a7ad982983f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/93722a2a9be5466cb3425a7ad982983f 2023-04-27 03:54:42,959 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/93722a2a9be5466cb3425a7ad982983f, entries=150, sequenceid=128, filesize=11.6 K 2023-04-27 03:54:42,960 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/4e49b5ea8cc142e3a0c83fd3c6931c28 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/4e49b5ea8cc142e3a0c83fd3c6931c28 2023-04-27 03:54:42,965 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/4e49b5ea8cc142e3a0c83fd3c6931c28, entries=150, sequenceid=128, filesize=11.6 K 2023-04-27 03:54:42,965 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/6cb3ee473c4144e580f8d546b01700bf as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/6cb3ee473c4144e580f8d546b01700bf 2023-04-27 03:54:42,971 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/6cb3ee473c4144e580f8d546b01700bf, entries=150, sequenceid=128, filesize=11.6 K 2023-04-27 03:54:42,972 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.86 KB/135028, heapSize ~374.97 KB/383968, currentSize=53.67 KB/54960 for e07ddca27cd19d46379b6939f22f78c7 in 475ms, sequenceid=128, compaction requested=true 2023-04-27 03:54:42,972 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:42,972 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:42,972 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:42,972 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:42,972 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,972 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:42,972 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:42,974 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,974 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:42,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:42,975 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,975 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:42,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:42,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:42,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:42,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:42,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:42,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46dac3f8[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:42,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:42,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,978 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:42,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:42,979 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,979 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:42,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:42,980 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:42,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:42,981 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:42,981 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:42,981 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:42,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:42,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:42,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:42,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:42,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:42,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:42,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:42,984 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:42,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:42,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:42,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:42,984 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:42,986 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,986 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:42,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,986 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:42,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:42,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:42,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:42,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:42,987 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:42,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:42,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:42,990 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:42,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,993 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,993 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:42,993 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:42,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:42,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:42,993 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:42,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:42,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:42,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:42,993 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:42,993 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:42,994 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:42,994 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,994 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:42,994 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:42,994 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:42,994 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:42,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:42,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:43,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:43,012 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e07ddca27cd19d46379b6939f22f78c7 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-27 03:54:43,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=A 2023-04-27 03:54:43,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:43,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=B 2023-04-27 03:54:43,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:43,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=C 2023-04-27 03:54:43,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:43,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567683024, max timestamp=1682567683043Num uniques 859; 2023-04-27 03:54:43,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:54:43,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567683024, max timestamp=1682567683043Num uniques 859; 2023-04-27 03:54:43,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:54:43,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567683024, max timestamp=1682567683043Num uniques 859; 2023-04-27 03:54:43,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:54:43,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:43,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567743061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:43,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567743063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:43,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567743064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:43,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567743064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,094 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:43,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:43,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:43,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:43,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:43,097 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:43,098 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:43,098 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:43,098 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:43,098 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,098 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,099 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:43,099 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:43,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:43,100 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,100 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:43,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:43,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,101 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:43,101 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:43,101 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:43,101 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:43,101 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:43,101 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:43,101 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:43,101 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:43,101 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:43,101 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:43,101 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:43,101 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:43,102 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:43,102 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:43,102 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,102 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:43,102 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:43,103 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,103 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:43,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:43,103 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,103 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:43,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:43,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:43,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:43,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:43,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26a09c7f[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:43,105 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:43,105 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,106 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,106 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:43,106 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:43,107 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:43,107 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,107 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:43,109 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,110 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:43,110 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:43,110 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:43,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:43,112 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:43,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:43,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:43,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:43,113 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:43,114 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:43,115 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,115 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:43,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:43,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:43,115 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:43,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,116 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:43,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,119 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,119 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,119 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,119 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,119 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:43,119 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:43,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:43,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:43,119 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,119 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,119 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:43,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:43,119 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:43,120 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:43,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:43,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:43,120 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:43,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:43,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567743165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:43,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567743167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:43,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567743169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:43,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567743169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,220 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:43,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:43,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:43,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:43,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:43,223 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:43,223 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:43,223 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:43,223 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:43,224 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,224 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,225 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,225 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:43,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:43,226 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,226 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:43,227 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:43,227 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:43,227 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:43,227 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:43,227 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:43,227 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:43,227 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:43,227 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:43,227 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:43,227 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:43,227 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:43,227 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:43,227 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:43,227 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,227 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:43,227 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:43,230 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:43,230 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:43,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:43,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:43,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bf59051[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:43,232 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:43,232 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,234 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:43,234 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:43,234 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:43,234 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,234 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:43,235 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,235 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:43,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:43,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:43,235 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:43,235 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:43,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:43,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:43,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:43,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:43,237 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:43,237 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:43,237 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:43,238 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:43,238 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:43,243 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,243 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:43,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:43,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,243 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:43,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:43,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:43,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:43,243 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:43,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,244 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,247 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:43,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:43,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:43,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,247 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:43,247 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:43,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:43,248 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:43,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:43,248 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:43,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:43,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:43,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:43,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:43,347 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:43,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:43,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:43,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:43,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:43,351 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:43,351 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:43,351 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:43,351 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:43,351 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,351 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,353 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,353 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:43,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:43,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:43,353 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,353 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:43,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:43,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:43,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:43,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:43,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:43,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:43,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:43,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:43,354 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:43,354 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:43,354 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:43,355 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:43,355 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:43,355 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:43,355 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,355 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:43,355 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:43,356 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,356 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:43,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:43,357 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,357 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:43,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:43,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:43,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:43,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:43,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fc6e1f3[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:43,358 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:43,358 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,360 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:43,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:43,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:43,361 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,361 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:43,362 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:43,362 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,362 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:43,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:43,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:43,362 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:43,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:43,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:43,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:43,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:43,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:43,365 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:43,365 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:43,365 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:43,365 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:43,365 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:43,366 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,366 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,366 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:43,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:43,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:43,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:43,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:43,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,367 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:43,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:43,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,368 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:43,371 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:43,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567743369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:43,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:43,371 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:43,371 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,372 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,372 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,372 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:43,372 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:43,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:43,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:43,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:43,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:43,372 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,372 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,372 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,372 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:43,372 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:43,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:43,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567743371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:43,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567743373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:43,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567743373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,421 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/c2e96bb6722b473e9a3627648c8c2370 2023-04-27 03:54:43,472 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:43,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:43,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:43,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:43,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:43,476 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:43,476 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:43,476 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:43,476 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:43,476 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,476 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,479 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,479 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:43,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:43,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:43,479 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,479 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:43,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:43,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:43,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:43,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:43,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:43,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:43,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:43,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:43,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:43,481 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:43,481 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:43,481 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:43,481 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:43,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:43,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:43,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:43,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:43,483 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,483 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:43,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:43,483 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,483 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:43,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:43,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:43,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:43,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:43,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f1338f0[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:43,485 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:43,485 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,486 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,486 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:43,486 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:43,486 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:43,486 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,486 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:43,488 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:43,488 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,488 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:43,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,488 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:43,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:43,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:43,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:43,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:43,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:43,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:43,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:43,490 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:43,490 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:43,490 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:43,490 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:43,490 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:43,492 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,492 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:43,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,493 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:43,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:43,493 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,494 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,496 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,496 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:43,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:43,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:43,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:43,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:43,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:43,497 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:43,497 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:43,497 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:43,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:43,598 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:43,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:43,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:43,600 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:43,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37876 deadline: 1682567743598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:43,602 DEBUG [Thread-1714] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., hostname=jenkins-hbase4.apache.org,40885,1682567618305, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 03:54:43,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:43,602 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:43,602 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:43,602 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:43,602 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:43,602 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,602 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,604 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:43,604 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:43,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:43,604 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,604 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:43,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:43,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:43,605 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:43,605 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:43,605 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:43,605 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:43,605 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:43,605 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:43,605 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:43,605 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:43,606 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:43,606 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:43,606 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:43,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:43,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:43,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:43,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:43,607 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,607 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:43,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:43,608 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,608 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:43,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:43,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:43,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:43,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:43,609 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:43,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d720209[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:43,609 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,611 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,611 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:43,611 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:43,611 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:43,611 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,611 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:43,612 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:43,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,613 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:43,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:43,613 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:43,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:43,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:43,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:43,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:43,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:43,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:43,615 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:43,615 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:43,615 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:43,615 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:43,615 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:43,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:43,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:43,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:43,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:43,617 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:43,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:43,619 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:43,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,624 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,624 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,624 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,624 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,624 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:43,624 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,624 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,625 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,624 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:43,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:43,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:43,625 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:43,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:43,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:43,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:43,625 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:43,625 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:43,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:43,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:43,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:43,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567743674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:43,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567743674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,679 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:43,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567743678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,679 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:43,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567743678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,725 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:43,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:43,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:43,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:43,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:43,728 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:43,728 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:43,728 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:43,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:43,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,730 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:43,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:43,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:43,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:43,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:43,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:43,732 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:43,732 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:43,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:43,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:43,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:43,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:43,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:43,732 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:43,732 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:43,732 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:43,732 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:43,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:43,733 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:43,733 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,733 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:43,733 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:43,734 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,734 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:43,735 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:43,735 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:43,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:43,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a3274ee[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:43,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:43,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,737 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,737 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:43,737 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:43,737 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:43,738 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,738 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:43,740 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:43,740 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:43,740 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,740 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:43,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:43,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:43,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:43,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:43,742 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:43,742 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:43,742 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:43,742 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:43,743 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:43,744 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,744 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:43,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:43,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,744 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:43,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:43,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:43,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:43,745 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:43,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,746 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:43,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,748 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:43,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:43,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:43,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:43,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:43,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:43,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,749 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:43,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:43,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:43,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,749 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:43,749 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:43,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,830 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/5e6c28db978342d798ca095e44cee049 2023-04-27 03:54:43,849 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:43,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:43,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:43,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:43,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:43,852 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:43,853 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:43,853 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:43,854 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:43,854 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,854 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,856 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,856 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:43,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:43,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:43,856 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,856 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:43,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:43,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,857 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:43,857 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:43,857 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:43,857 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:43,857 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:43,857 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:43,857 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:43,857 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:43,857 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:43,857 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:43,857 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:43,858 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:43,858 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:43,858 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:43,858 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,858 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:43,858 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:43,860 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,860 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:43,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:43,860 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,860 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:43,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:43,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3233db22[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:43,862 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:43,862 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,863 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,864 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,864 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:43,864 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:43,864 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:43,864 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:43,865 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:43,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:43,866 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:43,866 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:43,866 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:43,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:43,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:43,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:43,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:43,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:43,868 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:43,868 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:43,868 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:43,868 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:43,868 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:43,869 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:43,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:43,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:43,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:43,870 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:43,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:43,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:43,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,871 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,875 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,875 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:43,875 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:43,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:43,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:43,875 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:43,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:43,875 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:43,875 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:43,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:43,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:43,875 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,876 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:43,876 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:43,876 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,876 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,876 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,976 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:43,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:43,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:43,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:43,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:43,979 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:43,979 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:43,979 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:43,979 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:43,980 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,980 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,981 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,981 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:43,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:43,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:43,981 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,981 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:43,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:43,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:43,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:43,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:43,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:43,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:43,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:43,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:43,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:43,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:43,983 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:43,983 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:43,983 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:43,983 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:43,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:43,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:43,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,984 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:43,984 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:43,985 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:43,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:43,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:43,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:43,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:43,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4491b12c[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:43,987 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:43,987 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,988 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:43,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:43,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:43,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:43,989 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,989 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:43,994 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:43,994 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,994 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:43,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:43,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:43,994 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:43,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:43,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:43,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:43,996 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:43,996 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:43,996 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:43,996 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:43,996 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:43,998 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,998 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:43,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:43,998 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:43,998 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:43,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:44,000 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:44,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:44,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:44,002 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,002 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:44,002 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:44,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:44,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:44,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:44,002 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:44,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:44,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:44,002 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:44,002 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:44,002 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:44,002 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:44,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:44,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:44,002 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,002 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:44,002 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:44,002 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:44,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:44,102 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:44,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:44,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:44,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:44,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:44,105 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:44,105 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:44,105 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:44,105 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:44,106 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:44,106 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:44,110 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:44,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:44,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:44,111 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,111 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:44,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:44,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:44,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:44,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:44,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:44,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:44,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:44,112 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:44,112 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:44,112 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:44,112 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:44,112 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:44,112 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:44,112 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:44,112 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:44,112 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:44,112 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:44,112 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:44,112 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,112 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:44,112 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:44,114 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:44,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:44,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:44,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:44,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:44,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:44,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:44,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:44,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:44,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:44,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19ae91fd[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:44,116 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:44,116 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:44,117 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:44,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:44,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:44,117 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:44,117 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:44,117 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:44,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:44,119 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,119 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:44,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:44,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:44,119 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:44,119 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:44,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:44,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:44,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:44,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:44,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:44,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:44,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:44,121 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:44,121 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:44,121 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:44,121 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:44,121 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:44,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:44,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:44,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:44,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:44,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:44,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:44,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:44,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:44,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:44,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:44,124 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:44,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:44,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:44,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:44,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:44,125 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:44,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:44,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,128 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:44,128 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:44,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:44,128 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:44,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:44,128 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:44,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:44,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:44,128 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:44,128 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:44,128 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,128 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:44,128 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:44,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:44,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:44,128 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:44,128 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:44,128 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:44,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:44,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567744180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:44,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567744181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:44,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567744181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:44,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567744183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,229 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:44,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:44,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:44,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:44,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:44,231 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:44,232 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:44,232 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:44,232 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:44,232 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:44,232 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:44,234 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:44,234 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:44,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:44,235 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,235 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:44,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:44,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:44,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:44,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:44,235 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:44,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:44,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:44,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:44,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:44,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:44,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:44,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:44,236 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:44,236 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:44,236 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:44,236 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:44,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:44,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:44,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:44,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:44,238 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,238 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:44,238 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:44,238 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:44,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:44,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:44,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:44,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70520322[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:44,239 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:44,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:44,240 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/bff8cec9fa984e4b8242d74e07ccedc1 2023-04-27 03:54:44,242 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:44,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:44,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:44,242 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:44,242 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:44,242 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:44,242 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,242 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:44,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:44,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:44,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:44,244 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:44,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:44,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:44,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:44,245 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/c2e96bb6722b473e9a3627648c8c2370 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/c2e96bb6722b473e9a3627648c8c2370 2023-04-27 03:54:44,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:44,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:44,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:44,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:44,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:44,247 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:44,247 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:44,247 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:44,247 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:44,247 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:44,249 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/c2e96bb6722b473e9a3627648c8c2370, entries=150, sequenceid=142, filesize=11.8 K 2023-04-27 03:54:44,249 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:44,249 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:44,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:44,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:44,249 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:44,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:44,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:44,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:44,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:44,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:44,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:44,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:44,250 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/5e6c28db978342d798ca095e44cee049 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/5e6c28db978342d798ca095e44cee049 2023-04-27 03:54:44,250 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:44,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:44,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:44,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,251 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:44,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:44,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,254 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/5e6c28db978342d798ca095e44cee049, entries=150, sequenceid=142, filesize=11.8 K 2023-04-27 03:54:44,254 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,254 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:44,254 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:44,254 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,254 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:44,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:44,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:44,254 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:44,254 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:44,254 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:44,255 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/bff8cec9fa984e4b8242d74e07ccedc1 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/bff8cec9fa984e4b8242d74e07ccedc1 2023-04-27 03:54:44,258 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/bff8cec9fa984e4b8242d74e07ccedc1, entries=150, sequenceid=142, filesize=11.8 K 2023-04-27 03:54:44,258 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for e07ddca27cd19d46379b6939f22f78c7 in 1246ms, sequenceid=142, compaction requested=true 2023-04-27 03:54:44,258 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:44,259 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:44,259 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:44,259 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:44,259 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:44,259 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:44,259 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:44,259 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:44,259 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:44,259 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:44,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:44,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:44,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:44,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:44,260 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36014 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:44,260 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/A is initiating minor compaction (all files) 2023-04-27 03:54:44,260 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36014 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:44,260 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/A in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:44,260 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/B is initiating minor compaction (all files) 2023-04-27 03:54:44,260 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/bb9da04d8b684408b943cdcb7e5c77cd, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/93722a2a9be5466cb3425a7ad982983f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/c2e96bb6722b473e9a3627648c8c2370] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=35.2 K 2023-04-27 03:54:44,260 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/B in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:44,260 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/144e4278319d42b88af5042aab61a92f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/4e49b5ea8cc142e3a0c83fd3c6931c28, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/5e6c28db978342d798ca095e44cee049] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=35.2 K 2023-04-27 03:54:44,261 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting bb9da04d8b684408b943cdcb7e5c77cd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1682567681511 2023-04-27 03:54:44,261 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 144e4278319d42b88af5042aab61a92f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1682567681511 2023-04-27 03:54:44,261 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 93722a2a9be5466cb3425a7ad982983f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1682567681577 2023-04-27 03:54:44,261 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 4e49b5ea8cc142e3a0c83fd3c6931c28, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1682567681577 2023-04-27 03:54:44,261 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting c2e96bb6722b473e9a3627648c8c2370, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682567682705 2023-04-27 03:54:44,262 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 5e6c28db978342d798ca095e44cee049, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682567682705 2023-04-27 03:54:44,268 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#A#compaction#231 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:44,269 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#B#compaction#232 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:44,277 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/53c7eba76aca4be6b6114d5969c107dd as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/53c7eba76aca4be6b6114d5969c107dd 2023-04-27 03:54:44,281 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/B of e07ddca27cd19d46379b6939f22f78c7 into 53c7eba76aca4be6b6114d5969c107dd(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:44,281 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:44,281 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/B, priority=13, startTime=1682567684259; duration=0sec 2023-04-27 03:54:44,281 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:44,281 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 03:54:44,282 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47832 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 03:54:44,282 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/C is initiating minor compaction (all files) 2023-04-27 03:54:44,282 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/C in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:44,282 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/e4630e195f424a30a4b8d6298320d6f5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/27a6125fd7d9416596dc8e22fb587295, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/6cb3ee473c4144e580f8d546b01700bf, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/bff8cec9fa984e4b8242d74e07ccedc1] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=46.7 K 2023-04-27 03:54:44,282 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting e4630e195f424a30a4b8d6298320d6f5, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1682567680894 2023-04-27 03:54:44,283 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 27a6125fd7d9416596dc8e22fb587295, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1682567681511 2023-04-27 03:54:44,283 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 6cb3ee473c4144e580f8d546b01700bf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1682567681577 2023-04-27 03:54:44,283 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting bff8cec9fa984e4b8242d74e07ccedc1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682567682705 2023-04-27 03:54:44,289 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#C#compaction#233 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:44,359 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:44,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:44,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:44,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:44,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:44,362 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:44,362 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:44,362 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:44,362 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:44,363 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:44,363 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:44,365 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:44,365 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:44,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:44,366 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:44,366 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:44,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:44,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:44,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:44,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:44,366 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:44,366 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:44,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:44,367 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:44,367 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:44,367 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:44,367 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:44,367 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:44,367 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:44,367 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:44,367 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e07ddca27cd19d46379b6939f22f78c7 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-27 03:54:44,367 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=A 2023-04-27 03:54:44,367 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:44,367 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=B 2023-04-27 03:54:44,367 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:44,367 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=C 2023-04-27 03:54:44,367 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:44,678 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/301cb37987bc4d708d855775b634ea1d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/301cb37987bc4d708d855775b634ea1d 2023-04-27 03:54:44,682 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/A of e07ddca27cd19d46379b6939f22f78c7 into 301cb37987bc4d708d855775b634ea1d(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:44,682 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:44,682 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/A, priority=13, startTime=1682567684259; duration=0sec 2023-04-27 03:54:44,682 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:44,697 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/70ed89c647bb465da4d5b7060533571f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/70ed89c647bb465da4d5b7060533571f 2023-04-27 03:54:44,701 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/C of e07ddca27cd19d46379b6939f22f78c7 into 70ed89c647bb465da4d5b7060533571f(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:44,701 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:44,701 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/C, priority=12, startTime=1682567684259; duration=0sec 2023-04-27 03:54:44,701 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:44,778 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/d70e4141fb61492f934ab6df75dcdfb6 2023-04-27 03:54:44,789 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/611d1ca178ea4823a9eb680bf06e4de8 2023-04-27 03:54:45,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:45,189 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:45,199 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/78b300159c6c435f913fe244b6c7756e 2023-04-27 03:54:45,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:45,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567745196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,204 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/d70e4141fb61492f934ab6df75dcdfb6 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/d70e4141fb61492f934ab6df75dcdfb6 2023-04-27 03:54:45,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:45,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567745200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:45,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567745201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:45,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567745201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,207 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/d70e4141fb61492f934ab6df75dcdfb6, entries=150, sequenceid=178, filesize=11.8 K 2023-04-27 03:54:45,208 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/611d1ca178ea4823a9eb680bf06e4de8 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/611d1ca178ea4823a9eb680bf06e4de8 2023-04-27 03:54:45,211 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/611d1ca178ea4823a9eb680bf06e4de8, entries=150, sequenceid=178, filesize=11.8 K 2023-04-27 03:54:45,212 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/78b300159c6c435f913fe244b6c7756e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/78b300159c6c435f913fe244b6c7756e 2023-04-27 03:54:45,215 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/78b300159c6c435f913fe244b6c7756e, entries=150, sequenceid=178, filesize=11.8 K 2023-04-27 03:54:45,216 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=67.09 KB/68700 for e07ddca27cd19d46379b6939f22f78c7 in 849ms, sequenceid=178, compaction requested=false 2023-04-27 03:54:45,216 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:45,216 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:45,216 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:45,216 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:45,216 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,216 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:45,216 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:45,218 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,218 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:45,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:45,218 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,218 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:45,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:45,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:45,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:45,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:45,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c142cc1[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:45,220 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:45,220 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,221 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,221 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:45,221 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:45,221 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:45,221 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,221 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:45,223 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,223 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:45,223 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:45,223 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:45,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:45,225 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:45,225 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:45,225 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:45,225 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:45,225 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:45,228 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,228 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,228 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:45,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:45,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:45,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:45,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:45,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:45,228 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:45,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,230 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:45,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,232 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,232 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,232 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:45,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:45,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:45,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:45,233 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:45,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:45,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:45,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:45,233 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:45,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,233 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:45,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:45,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:45,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:45,303 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e07ddca27cd19d46379b6939f22f78c7 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-27 03:54:45,303 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=A 2023-04-27 03:54:45,303 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:45,304 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=B 2023-04-27 03:54:45,304 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:45,304 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=C 2023-04-27 03:54:45,304 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:45,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:45,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567745319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:45,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567745321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:45,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567745323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:45,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567745323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,333 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:45,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:45,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:45,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:45,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:45,336 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:45,336 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:45,336 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:45,337 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:45,337 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,337 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,338 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,338 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:45,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:45,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:45,339 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,339 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:45,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:45,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:45,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:45,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:45,340 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:45,340 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:45,340 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:45,340 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:45,340 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:45,340 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:45,340 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:45,340 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:45,340 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:45,340 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:45,340 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:45,340 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,340 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:45,340 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:45,345 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,345 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:45,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:45,346 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,346 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:45,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:45,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:45,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:45,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:45,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15c5febe[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:45,347 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:45,348 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,349 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,349 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:45,349 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:45,349 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:45,349 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,349 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:45,350 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:45,350 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:45,350 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,350 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:45,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:45,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:45,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:45,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:45,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:45,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:45,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:45,353 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:45,353 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:45,353 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:45,353 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:45,353 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:45,354 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,354 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:45,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:45,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,354 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:45,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:45,355 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:45,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:45,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:45,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:45,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,358 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,358 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,358 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:45,358 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:45,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:45,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:45,358 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:45,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:45,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:45,358 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,359 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,359 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,359 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,359 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,359 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:45,359 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:45,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:45,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:45,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:45,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:45,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:45,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567745424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567745424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:45,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567745426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:45,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567745426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,459 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:45,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:45,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:45,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:45,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:45,462 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:45,462 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:45,462 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:45,462 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:45,463 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,463 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,464 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,464 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:45,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:45,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:45,464 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,465 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:45,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:45,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:45,465 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:45,465 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:45,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:45,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:45,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:45,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:45,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:45,466 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:45,466 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:45,466 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:45,466 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:45,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:45,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:45,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:45,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:45,468 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,468 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:45,468 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,468 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:45,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:45,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:45,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:45,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7889a482[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:45,470 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:45,470 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,471 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,471 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:45,471 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:45,471 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:45,471 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,471 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:45,473 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:45,473 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:45,473 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,473 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:45,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:45,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:45,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:45,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:45,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:45,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:45,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:45,476 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:45,476 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:45,476 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:45,476 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:45,476 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:45,477 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,477 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:45,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:45,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,477 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:45,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:45,478 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:45,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:45,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:45,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,480 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:45,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,482 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:45,482 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:45,482 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:45,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:45,482 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:45,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:45,482 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:45,482 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,482 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,482 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,482 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:45,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:45,482 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:45,482 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,482 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:45,583 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:45,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:45,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:45,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:45,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:45,586 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:45,586 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:45,586 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:45,586 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:45,586 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,586 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,588 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,588 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:45,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:45,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:45,588 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,588 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:45,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:45,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:45,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:45,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:45,589 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:45,589 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:45,589 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:45,589 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:45,589 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:45,590 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:45,590 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:45,590 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:45,590 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:45,590 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:45,590 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:45,590 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,590 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:45,590 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:45,592 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,592 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:45,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:45,593 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:45,593 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:45,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:45,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:45,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:45,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1822f3a2[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:45,594 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:45,594 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,595 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,595 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:45,595 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:45,596 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:45,595 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,596 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:45,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:45,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:45,597 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:45,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:45,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:45,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:45,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:45,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:45,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:45,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:45,600 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:45,600 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:45,600 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:45,600 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:45,600 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:45,601 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,601 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,601 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:45,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:45,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:45,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:45,602 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:45,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:45,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:45,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,604 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:45,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:45,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:45,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:45,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,612 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:45,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:45,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:45,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:45,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:45,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,613 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:45,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:45,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:45,613 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:45,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:45,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567745627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:45,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567745629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:45,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567745629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:45,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567745629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,713 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:45,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:45,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:45,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:45,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:45,717 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:45,717 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:45,717 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:45,717 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:45,717 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,718 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,718 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/9607be49a68246bfb48d22ee71e0de96 2023-04-27 03:54:45,719 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,719 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:45,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:45,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:45,720 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,720 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:45,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:45,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:45,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:45,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:45,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:45,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:45,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:45,721 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:45,721 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:45,721 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:45,721 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:45,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:45,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:45,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:45,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:45,724 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,724 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:45,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:45,724 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,724 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:45,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:45,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:45,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:45,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:45,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@652db643[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:45,726 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:45,726 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,727 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,727 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:45,727 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:45,727 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:45,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,728 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:45,732 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:45,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:45,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:45,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:45,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:45,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:45,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:45,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:45,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:45,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:45,734 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:45,734 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:45,734 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:45,735 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:45,735 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:45,736 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,736 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:45,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,736 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:45,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:45,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:45,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:45,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:45,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:45,738 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:45,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,741 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,741 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,741 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,741 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:45,741 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:45,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:45,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:45,741 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:45,741 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,741 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,741 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,741 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:45,741 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:45,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:45,741 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:45,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:45,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:45,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:45,842 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:45,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:45,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:45,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:45,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:45,845 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:45,845 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:45,845 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:45,845 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:45,845 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,845 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,847 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,847 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:45,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:45,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:45,847 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,847 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:45,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:45,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:45,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:45,849 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:45,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:45,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:45,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:45,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:45,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:45,849 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:45,849 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:45,849 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:45,849 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:45,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:45,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:45,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:45,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:45,852 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,852 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:45,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:45,852 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,852 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:45,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:45,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:45,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:45,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:45,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f279456[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:45,854 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:45,854 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,855 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:45,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:45,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:45,855 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,855 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:45,856 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:45,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:45,857 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:45,857 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:45,857 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:45,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:45,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:45,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:45,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:45,859 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:45,859 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:45,859 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:45,859 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:45,859 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:45,861 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,861 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:45,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:45,861 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:45,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,861 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:45,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:45,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:45,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:45,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,863 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:45,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,866 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,866 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:45,866 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,866 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:45,866 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:45,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:45,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:45,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:45,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:45,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:45,866 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,867 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,867 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,867 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,867 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,867 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:45,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:45,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:45,867 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:45,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:45,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567745931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:45,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567745931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:45,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567745932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:45,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567745933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,967 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:45,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:45,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:45,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:45,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:45,970 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:45,970 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:45,970 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:45,970 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:45,971 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,971 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,972 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:45,972 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:45,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:45,973 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,973 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:45,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:45,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,974 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:45,974 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:45,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:45,974 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:45,974 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:45,974 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:45,974 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:45,974 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:45,974 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:45,974 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:45,974 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:45,975 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:45,975 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:45,975 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:45,975 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,975 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:45,975 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:45,976 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,977 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:45,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:45,977 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,977 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:45,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:45,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:45,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:45,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:45,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6abf9684[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:45,978 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:45,978 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,980 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:45,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:45,980 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,980 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:45,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:45,982 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:45,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:45,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:45,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:45,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:45,985 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:45,985 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:45,985 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:45,985 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:45,985 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:45,987 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:45,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:45,987 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,987 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:45,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:45,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:45,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:45,987 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:45,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,988 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:45,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,991 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,991 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,991 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:45,991 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:45,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:45,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:45,991 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:45,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:45,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:45,991 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:45,991 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:45,992 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,992 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:45,992 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:45,992 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:45,992 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:45,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:45,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:46,092 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:46,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:46,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:46,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:46,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:46,095 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:46,095 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:46,095 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:46,095 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:46,096 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,096 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:46,097 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:46,097 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:46,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:46,098 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,098 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:46,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:46,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:46,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:46,099 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:46,099 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:46,099 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:46,099 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:46,099 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:46,099 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:46,099 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:46,099 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:46,099 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:46,099 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:46,099 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:46,099 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:46,100 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:46,100 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:46,100 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,100 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:46,100 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:46,103 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,103 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:46,103 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,103 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:46,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:46,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:46,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:46,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@658fd792[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:46,104 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:46,104 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,106 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,106 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:46,106 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:46,106 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,106 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:46,106 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:46,108 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:46,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:46,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:46,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:46,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:46,110 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:46,110 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:46,110 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:46,110 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:46,110 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:46,117 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,117 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:46,117 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:46,118 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:46,119 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:46,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:46,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:46,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:46,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:46,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:46,124 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:46,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:46,124 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:46,124 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:46,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:46,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,129 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/dab540ded83c4dc5aa5207faf197069f 2023-04-27 03:54:46,224 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:46,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:46,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:46,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:46,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:46,228 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:46,228 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:46,228 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:46,228 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:46,228 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,228 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:46,231 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:46,231 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:46,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:46,231 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,231 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:46,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:46,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:46,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:46,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:46,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:46,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:46,232 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:46,232 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:46,232 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:46,232 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:46,232 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:46,233 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:46,233 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:46,233 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:46,233 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:46,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:46,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:46,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:46,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:46,235 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:46,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:46,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:46,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:46,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d98eace[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:46,237 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:46,237 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,240 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,240 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:46,240 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:46,240 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:46,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:46,243 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:46,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:46,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:46,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:46,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:46,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:46,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:46,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:46,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:46,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:46,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:46,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:46,245 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:46,245 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:46,245 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:46,245 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:46,245 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:46,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:46,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:46,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:46,248 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:46,249 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,251 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,251 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:46,251 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:46,251 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:46,251 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,251 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:46,251 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:46,251 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,251 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:46,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:46,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:46,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:46,252 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,252 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:46,252 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:46,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:46,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:46,352 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:46,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:46,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:46,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:46,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:46,355 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:46,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:46,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:46,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:46,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:46,357 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,357 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:46,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:46,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:46,358 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,358 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:46,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:46,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:46,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:46,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:46,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:46,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:46,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:46,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:46,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:46,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:46,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:46,359 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:46,359 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:46,359 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:46,359 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:46,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:46,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:46,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:46,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:46,362 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:46,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:46,363 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,363 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:46,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:46,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:46,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:46,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:46,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@efe560f[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:46,364 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:46,364 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,365 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,366 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:46,366 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:46,366 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:46,366 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,366 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:46,367 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:46,367 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,367 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:46,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:46,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:46,367 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:46,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:46,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:46,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:46,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:46,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:46,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:46,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:46,370 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:46,370 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:46,370 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:46,370 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:46,370 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:46,372 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,372 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,372 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:46,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:46,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:46,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,373 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,374 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:46,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:46,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,377 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,378 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:46,378 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:46,378 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:46,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:46,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:46,378 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:46,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:46,378 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,378 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:46,378 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:46,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:46,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:46,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:46,378 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,378 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,378 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:46,379 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:46,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:46,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567746434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:46,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567746436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:46,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567746436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:46,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567746446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,479 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:46,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:46,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:46,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:46,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:46,482 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:46,482 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:46,482 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:46,482 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:46,483 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,483 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:46,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:46,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:46,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:46,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:46,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:46,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:46,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:46,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:46,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:46,485 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:46,485 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:46,485 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:46,486 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:46,486 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:46,486 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:46,486 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:46,486 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:46,486 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:46,486 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:46,486 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:46,486 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,486 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:46,486 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:46,488 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,488 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:46,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:46,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:46,488 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,488 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:46,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:46,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:46,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:46,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:46,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18e3eae2[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:46,489 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:46,489 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,491 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,491 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:46,491 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,491 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:46,491 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:46,491 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:46,493 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,493 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:46,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:46,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:46,493 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:46,494 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:46,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:46,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:46,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:46,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:46,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:46,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:46,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:46,496 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:46,496 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:46,496 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:46,496 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:46,496 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:46,498 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,498 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:46,498 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:46,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:46,499 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:46,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:46,500 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:46,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:46,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,506 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:46,506 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:46,506 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:46,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:46,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:46,506 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:46,506 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:46,506 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:46,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:46,506 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:46,506 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:46,506 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,506 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,506 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:46,506 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:46,539 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/a79142253d9349598bc93edd7c429794 2023-04-27 03:54:46,544 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/9607be49a68246bfb48d22ee71e0de96 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/9607be49a68246bfb48d22ee71e0de96 2023-04-27 03:54:46,547 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/9607be49a68246bfb48d22ee71e0de96, entries=200, sequenceid=194, filesize=14.1 K 2023-04-27 03:54:46,548 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/dab540ded83c4dc5aa5207faf197069f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/dab540ded83c4dc5aa5207faf197069f 2023-04-27 03:54:46,551 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/dab540ded83c4dc5aa5207faf197069f, entries=150, sequenceid=194, filesize=11.8 K 2023-04-27 03:54:46,552 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/a79142253d9349598bc93edd7c429794 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/a79142253d9349598bc93edd7c429794 2023-04-27 03:54:46,555 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/a79142253d9349598bc93edd7c429794, entries=150, sequenceid=194, filesize=11.8 K 2023-04-27 03:54:46,556 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for e07ddca27cd19d46379b6939f22f78c7 in 1253ms, sequenceid=194, compaction requested=true 2023-04-27 03:54:46,556 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:46,556 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:46,556 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:46,556 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:46,556 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:46,556 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:46,557 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38806 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:46,557 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/A is initiating minor compaction (all files) 2023-04-27 03:54:46,557 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:46,557 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/A in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:46,557 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/B is initiating minor compaction (all files) 2023-04-27 03:54:46,557 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/301cb37987bc4d708d855775b634ea1d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/d70e4141fb61492f934ab6df75dcdfb6, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/9607be49a68246bfb48d22ee71e0de96] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=37.9 K 2023-04-27 03:54:46,557 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/B in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:46,557 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/53c7eba76aca4be6b6114d5969c107dd, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/611d1ca178ea4823a9eb680bf06e4de8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/dab540ded83c4dc5aa5207faf197069f] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=35.6 K 2023-04-27 03:54:46,557 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 301cb37987bc4d708d855775b634ea1d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682567682705 2023-04-27 03:54:46,558 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 53c7eba76aca4be6b6114d5969c107dd, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682567682705 2023-04-27 03:54:46,558 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting d70e4141fb61492f934ab6df75dcdfb6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1682567683055 2023-04-27 03:54:46,558 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 611d1ca178ea4823a9eb680bf06e4de8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1682567683055 2023-04-27 03:54:46,558 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 9607be49a68246bfb48d22ee71e0de96, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1682567685199 2023-04-27 03:54:46,558 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting dab540ded83c4dc5aa5207faf197069f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1682567685199 2023-04-27 03:54:46,567 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#B#compaction#241 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:46,568 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#A#compaction#240 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:46,606 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:46,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:46,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:46,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:46,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:46,609 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:46,609 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:46,609 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:46,609 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:46,610 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,610 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:46,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:46,612 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:46,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:46,612 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:46,612 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:46,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:46,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:46,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:46,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:46,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:46,613 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:46,614 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:46,614 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:46,614 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:46,614 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:46,614 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:46,614 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:46,614 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e07ddca27cd19d46379b6939f22f78c7 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-27 03:54:46,614 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=A 2023-04-27 03:54:46,614 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:46,614 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=B 2023-04-27 03:54:46,614 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:46,614 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=C 2023-04-27 03:54:46,615 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:46,981 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/352e55b79d3b489ea37ce6f3d5e47894 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/352e55b79d3b489ea37ce6f3d5e47894 2023-04-27 03:54:46,981 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/6ab831af0bba4d2ab74cae0d68630b7a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/6ab831af0bba4d2ab74cae0d68630b7a 2023-04-27 03:54:46,985 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/B of e07ddca27cd19d46379b6939f22f78c7 into 352e55b79d3b489ea37ce6f3d5e47894(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:46,985 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/A of e07ddca27cd19d46379b6939f22f78c7 into 6ab831af0bba4d2ab74cae0d68630b7a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:46,986 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:46,986 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/A, priority=13, startTime=1682567686556; duration=0sec 2023-04-27 03:54:46,985 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:46,986 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/B, priority=13, startTime=1682567686556; duration=0sec 2023-04-27 03:54:46,986 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:46,986 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:46,986 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:46,987 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:46,987 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/C is initiating minor compaction (all files) 2023-04-27 03:54:46,987 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/C in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:46,987 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/70ed89c647bb465da4d5b7060533571f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/78b300159c6c435f913fe244b6c7756e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/a79142253d9349598bc93edd7c429794] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=35.5 K 2023-04-27 03:54:46,987 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 70ed89c647bb465da4d5b7060533571f, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682567682705 2023-04-27 03:54:46,988 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 78b300159c6c435f913fe244b6c7756e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1682567683055 2023-04-27 03:54:46,988 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting a79142253d9349598bc93edd7c429794, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1682567685199 2023-04-27 03:54:46,993 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#C#compaction#243 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:47,022 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/ead45b63cc24402e8e914283c4e6f387 2023-04-27 03:54:47,404 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/c6d1c350afd042dbae0fdb443c1e6228 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/c6d1c350afd042dbae0fdb443c1e6228 2023-04-27 03:54:47,408 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/C of e07ddca27cd19d46379b6939f22f78c7 into c6d1c350afd042dbae0fdb443c1e6228(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:47,409 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:47,409 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/C, priority=13, startTime=1682567686556; duration=0sec 2023-04-27 03:54:47,409 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:47,432 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/9ecdb14462f04e73968bf27d326581d3 2023-04-27 03:54:47,444 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/294e63c233124474a5ed156440dc3ba1 2023-04-27 03:54:47,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:47,447 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:47,449 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/ead45b63cc24402e8e914283c4e6f387 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/ead45b63cc24402e8e914283c4e6f387 2023-04-27 03:54:47,454 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/ead45b63cc24402e8e914283c4e6f387, entries=150, sequenceid=213, filesize=11.8 K 2023-04-27 03:54:47,455 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/9ecdb14462f04e73968bf27d326581d3 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/9ecdb14462f04e73968bf27d326581d3 2023-04-27 03:54:47,459 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/9ecdb14462f04e73968bf27d326581d3, entries=150, sequenceid=213, filesize=11.8 K 2023-04-27 03:54:47,460 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/294e63c233124474a5ed156440dc3ba1 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/294e63c233124474a5ed156440dc3ba1 2023-04-27 03:54:47,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:47,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567747457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:47,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567747459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:47,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567747459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:47,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567747460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,463 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/294e63c233124474a5ed156440dc3ba1, entries=150, sequenceid=213, filesize=11.8 K 2023-04-27 03:54:47,464 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for e07ddca27cd19d46379b6939f22f78c7 in 850ms, sequenceid=213, compaction requested=false 2023-04-27 03:54:47,464 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:47,464 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:47,464 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:47,464 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:47,464 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,464 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:47,464 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:47,466 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:47,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:47,467 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,467 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:47,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:47,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:47,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:47,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:47,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cb9d2b[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:47,468 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:47,468 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,470 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,470 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:47,471 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:47,471 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:47,471 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,471 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:47,472 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:47,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:47,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:47,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:47,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:47,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:47,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:47,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:47,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:47,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:47,475 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:47,475 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:47,475 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:47,475 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:47,476 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:47,478 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,478 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:47,478 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:47,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:47,480 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:47,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,481 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:47,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,485 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:47,485 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:47,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:47,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:47,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:47,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:47,486 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,486 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:47,486 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:47,486 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,486 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:47,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:47,486 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:47,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:47,486 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:47,486 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:47,486 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,486 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:47,564 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e07ddca27cd19d46379b6939f22f78c7 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-27 03:54:47,564 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=A 2023-04-27 03:54:47,564 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:47,564 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=B 2023-04-27 03:54:47,564 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:47,564 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=C 2023-04-27 03:54:47,564 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:47,578 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/a50e26ef347f49f1810f9b2cc657ca71 2023-04-27 03:54:47,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:47,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567747577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:47,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567747578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:47,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567747579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:47,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567747580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,586 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:47,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:47,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:47,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:47,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:47,590 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:47,590 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:47,590 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:47,590 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:47,590 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,590 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:47,591 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:47,592 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:47,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:47,592 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,592 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:47,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:47,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:47,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:47,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:47,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:47,593 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:47,593 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:47,593 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:47,593 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:47,593 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:47,593 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:47,593 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:47,593 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:47,593 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:47,593 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:47,593 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:47,593 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,594 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:47,594 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:47,596 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,596 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:47,596 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,596 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:47,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:47,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@292fc906[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:47,598 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:47,598 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,600 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,601 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:47,601 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:47,601 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:47,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:47,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:47,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:47,602 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:47,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:47,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:47,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:47,605 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:47,605 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:47,605 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:47,605 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:47,605 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:47,607 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,607 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,607 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:47,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:47,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:47,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:47,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:47,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:47,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,609 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,610 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:47,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:47,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:47,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:47,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:47,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:47,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:47,612 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:47,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:47,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:47,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:47,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,613 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:47,613 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:47,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:47,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37876 deadline: 1682567747632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,635 DEBUG [Thread-1714] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8181 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., hostname=jenkins-hbase4.apache.org,40885,1682567618305, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 03:54:47,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:47,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567747681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:47,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567747681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:47,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567747683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:47,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567747684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,713 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:47,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:47,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:47,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:47,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:47,716 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:47,716 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:47,716 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:47,716 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:47,717 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,717 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:47,719 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,719 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:47,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:47,720 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,720 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:47,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:47,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:47,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:47,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:47,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:47,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:47,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:47,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:47,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:47,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:47,721 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:47,721 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:47,722 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:47,722 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:47,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:47,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:47,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:47,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:47,723 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:47,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:47,724 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,724 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:47,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:47,725 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:47,725 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@559dac02[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:47,726 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,727 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:47,727 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:47,727 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:47,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:47,728 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,728 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:47,728 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:47,728 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:47,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:47,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:47,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:47,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:47,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:47,730 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:47,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:47,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:47,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:47,731 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:47,733 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,733 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:47,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,733 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:47,734 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,735 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:47,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,737 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,737 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:47,737 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:47,738 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:47,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:47,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:47,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:47,738 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:47,738 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:47,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:47,738 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,738 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:47,738 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,738 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:47,738 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,738 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,838 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:47,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:47,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:47,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:47,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:47,842 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:47,842 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:47,842 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:47,842 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:47,842 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,842 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:47,846 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:47,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:47,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:47,847 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,847 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,847 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:47,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:47,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:47,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:47,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:47,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:47,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:47,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:47,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:47,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:47,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:47,848 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:47,848 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:47,848 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:47,849 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:47,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:47,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:47,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:47,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:47,851 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,851 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:47,851 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,851 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:47,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:47,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:47,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:47,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@522ef8a9[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:47,853 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:47,853 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,854 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,854 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,854 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:47,854 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:47,854 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:47,854 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:47,856 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:47,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:47,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:47,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:47,858 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:47,858 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:47,858 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:47,858 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:47,859 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:47,860 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,860 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,860 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:47,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:47,861 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:47,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,862 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,869 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:47,869 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,869 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:47,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:47,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:47,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:47,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:47,869 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:47,869 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:47,869 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:47,869 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:47,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,870 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:47,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,870 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:47,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:47,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:47,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567747884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:47,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567747884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:47,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567747885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:47,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567747887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,970 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:47,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:47,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:47,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:47,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:47,973 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:47,973 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:47,973 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:47,973 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:47,974 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,974 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:47,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,975 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:47,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:47,976 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,976 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:47,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:47,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:47,977 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:47,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:47,977 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:47,977 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:47,977 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:47,977 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:47,977 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:47,977 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:47,977 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:47,977 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:47,977 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:47,977 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:47,978 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:47,978 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,978 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:47,978 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:47,979 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:47,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:47,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:47,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:47,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:47,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:47,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:47,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d530502[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:47,981 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:47,981 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,982 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:47,983 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,983 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:47,985 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:47,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:47,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:47,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:47,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:47,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:47,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:47,987 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:47,987 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:47,987 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:47,987 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:47,988 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:47,989 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,989 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,989 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:47,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,990 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/2c877e982f0041499903a9cd31fd27c2 2023-04-27 03:54:47,990 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:47,992 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,995 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:47,995 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,995 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:47,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:47,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:47,996 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:47,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:47,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:47,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:47,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:47,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:47,996 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:47,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:47,996 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:47,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:47,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:48,096 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:48,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:48,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:48,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:48,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:48,100 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:48,100 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:48,100 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:48,100 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:48,100 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,100 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:48,102 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:48,102 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:48,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:48,102 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,102 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:48,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:48,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:48,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:48,103 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:48,103 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:48,103 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:48,103 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:48,103 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:48,104 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:48,104 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:48,104 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:48,104 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:48,104 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:48,104 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:48,104 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:48,104 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:48,104 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,104 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:48,104 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:48,109 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:48,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:48,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:48,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:48,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:48,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:48,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:48,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:48,111 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:48,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a638a57[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:48,111 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:48,112 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:48,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:48,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:48,113 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:48,113 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:48,113 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:48,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:48,114 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:48,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:48,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:48,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:48,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:48,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:48,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:48,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:48,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:48,117 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:48,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:48,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:48,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:48,117 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:48,120 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,120 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:48,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:48,120 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:48,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:48,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:48,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:48,121 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:48,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:48,122 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,125 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,125 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:48,125 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:48,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:48,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:48,125 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:48,125 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,125 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:48,126 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:48,126 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,125 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:48,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:48,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:48,125 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:48,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:48,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:48,126 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:48,126 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:48,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:48,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567748188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:48,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567748188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:48,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567748188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:48,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567748190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,226 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:48,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:48,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:48,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:48,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:48,229 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:48,229 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:48,229 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:48,229 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:48,229 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,229 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:48,232 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:48,232 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:48,232 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,232 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:48,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:48,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:48,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:48,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:48,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:48,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:48,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:48,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:48,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:48,233 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:48,233 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:48,233 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:48,233 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:48,234 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:48,234 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:48,234 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,234 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:48,234 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:48,239 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:48,239 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:48,239 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:48,239 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:48,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:48,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:48,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:48,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:48,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62a6d0a5[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:48,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:48,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:48,244 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:48,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:48,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:48,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:48,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:48,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:48,244 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,244 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:48,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,247 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:48,247 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:48,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:48,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:48,247 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:48,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:48,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:48,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:48,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:48,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:48,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:48,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:48,249 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:48,249 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:48,249 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:48,249 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:48,250 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:48,253 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,253 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:48,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,253 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:48,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:48,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:48,254 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:48,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:48,255 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,257 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:48,257 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:48,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:48,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:48,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:48,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:48,257 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,257 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:48,258 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:48,258 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:48,258 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:48,258 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:48,258 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:48,258 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,258 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:48,258 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:48,358 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:48,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:48,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:48,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:48,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:48,361 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:48,361 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:48,362 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:48,362 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:48,362 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,362 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:48,369 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:48,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:48,369 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:48,370 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,370 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:48,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:48,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:48,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:48,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:48,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:48,371 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:48,371 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:48,371 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:48,371 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:48,371 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:48,371 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:48,371 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:48,371 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:48,371 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:48,371 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:48,371 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:48,371 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,371 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:48,371 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:48,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:48,373 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:48,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:48,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:48,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:48,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:48,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:48,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:48,374 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:48,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47e897e8[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:48,375 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:48,376 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:48,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:48,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:48,376 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:48,376 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:48,376 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:48,376 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,376 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:48,377 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,377 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:48,377 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:48,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:48,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:48,377 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:48,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:48,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:48,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:48,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:48,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:48,379 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:48,380 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:48,380 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:48,380 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:48,380 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:48,382 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,382 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:48,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:48,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,382 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:48,383 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:48,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,384 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:48,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,386 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,386 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:48,386 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:48,386 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:48,386 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:48,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:48,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:48,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:48,386 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:48,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:48,386 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:48,386 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:48,386 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:48,386 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,386 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:48,386 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:48,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:48,401 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/272ff850ed994a6690a23be09a31f269 2023-04-27 03:54:48,405 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/a50e26ef347f49f1810f9b2cc657ca71 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/a50e26ef347f49f1810f9b2cc657ca71 2023-04-27 03:54:48,408 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/a50e26ef347f49f1810f9b2cc657ca71, entries=150, sequenceid=230, filesize=11.8 K 2023-04-27 03:54:48,409 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/2c877e982f0041499903a9cd31fd27c2 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/2c877e982f0041499903a9cd31fd27c2 2023-04-27 03:54:48,412 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/2c877e982f0041499903a9cd31fd27c2, entries=150, sequenceid=230, filesize=11.8 K 2023-04-27 03:54:48,413 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/272ff850ed994a6690a23be09a31f269 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/272ff850ed994a6690a23be09a31f269 2023-04-27 03:54:48,416 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/272ff850ed994a6690a23be09a31f269, entries=150, sequenceid=230, filesize=11.8 K 2023-04-27 03:54:48,417 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for e07ddca27cd19d46379b6939f22f78c7 in 853ms, sequenceid=230, compaction requested=true 2023-04-27 03:54:48,417 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:48,417 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:48,417 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:48,417 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:48,417 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:48,417 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:48,418 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:48,418 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/A is initiating minor compaction (all files) 2023-04-27 03:54:48,418 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:48,418 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/A in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:48,418 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/B is initiating minor compaction (all files) 2023-04-27 03:54:48,418 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/6ab831af0bba4d2ab74cae0d68630b7a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/ead45b63cc24402e8e914283c4e6f387, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/a50e26ef347f49f1810f9b2cc657ca71] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=35.7 K 2023-04-27 03:54:48,418 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/B in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:48,418 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/352e55b79d3b489ea37ce6f3d5e47894, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/9ecdb14462f04e73968bf27d326581d3, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/2c877e982f0041499903a9cd31fd27c2] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=35.7 K 2023-04-27 03:54:48,419 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 6ab831af0bba4d2ab74cae0d68630b7a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1682567685199 2023-04-27 03:54:48,419 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 352e55b79d3b489ea37ce6f3d5e47894, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1682567685199 2023-04-27 03:54:48,419 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting ead45b63cc24402e8e914283c4e6f387, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682567685308 2023-04-27 03:54:48,419 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 9ecdb14462f04e73968bf27d326581d3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682567685308 2023-04-27 03:54:48,419 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting a50e26ef347f49f1810f9b2cc657ca71, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682567687449 2023-04-27 03:54:48,419 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 2c877e982f0041499903a9cd31fd27c2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682567687449 2023-04-27 03:54:48,426 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#A#compaction#249 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:48,427 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#B#compaction#250 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:48,486 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:48,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:48,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:48,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:48,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:48,490 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:48,490 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:48,490 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:48,490 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:48,490 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,490 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:48,492 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:48,492 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:48,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:48,492 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,492 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:48,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:48,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:48,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:48,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:48,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:48,493 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:48,493 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:48,493 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:48,493 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:48,493 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:48,493 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:48,493 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:48,494 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e07ddca27cd19d46379b6939f22f78c7 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-27 03:54:48,494 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=A 2023-04-27 03:54:48,494 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:48,494 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=B 2023-04-27 03:54:48,494 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:48,494 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=C 2023-04-27 03:54:48,494 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:48,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:48,691 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:48,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:48,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567748700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:48,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567748702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:48,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567748703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:48,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567748703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:48,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567748804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:48,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567748807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:48,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567748808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:48,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567748808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:48,835 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/5ac1f73a478c439f95381e70e18a8a2b as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/5ac1f73a478c439f95381e70e18a8a2b 2023-04-27 03:54:48,837 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/0cbc82a5b4294ba68157c0676bfc02d5 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/0cbc82a5b4294ba68157c0676bfc02d5 2023-04-27 03:54:48,839 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/A of e07ddca27cd19d46379b6939f22f78c7 into 5ac1f73a478c439f95381e70e18a8a2b(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:48,840 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:48,840 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/A, priority=13, startTime=1682567688417; duration=0sec 2023-04-27 03:54:48,840 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:48,840 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:48,841 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:48,841 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/C is initiating minor compaction (all files) 2023-04-27 03:54:48,841 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/C in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:48,841 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/c6d1c350afd042dbae0fdb443c1e6228, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/294e63c233124474a5ed156440dc3ba1, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/272ff850ed994a6690a23be09a31f269] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=35.6 K 2023-04-27 03:54:48,842 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting c6d1c350afd042dbae0fdb443c1e6228, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1682567685199 2023-04-27 03:54:48,842 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/B of e07ddca27cd19d46379b6939f22f78c7 into 0cbc82a5b4294ba68157c0676bfc02d5(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:48,842 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:48,842 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/B, priority=13, startTime=1682567688417; duration=0sec 2023-04-27 03:54:48,842 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:48,842 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 294e63c233124474a5ed156440dc3ba1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682567685308 2023-04-27 03:54:48,842 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 272ff850ed994a6690a23be09a31f269, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682567687449 2023-04-27 03:54:48,849 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#C#compaction#252 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:48,901 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/52587078d9c244c28f2da9695e910002 2023-04-27 03:54:49,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:49,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567749008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:49,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567749010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:49,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567749010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:49,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37796 deadline: 1682567749011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,257 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/5c8836737685436f8354736f3911418a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/5c8836737685436f8354736f3911418a 2023-04-27 03:54:49,261 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/C of e07ddca27cd19d46379b6939f22f78c7 into 5c8836737685436f8354736f3911418a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:49,261 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:49,261 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/C, priority=13, startTime=1682567688417; duration=0sec 2023-04-27 03:54:49,261 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:49,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:49,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37856 deadline: 1682567749310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,311 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/959807de1c90480c9c48f0aff3f7d73f 2023-04-27 03:54:49,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:49,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567749312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:49,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37840 deadline: 1682567749314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:49,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567749316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,722 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/45a43f0c7d0b48e0ae25cbd999e8ced7 2023-04-27 03:54:49,726 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/52587078d9c244c28f2da9695e910002 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/52587078d9c244c28f2da9695e910002 2023-04-27 03:54:49,730 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/52587078d9c244c28f2da9695e910002, entries=150, sequenceid=248, filesize=11.8 K 2023-04-27 03:54:49,731 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/959807de1c90480c9c48f0aff3f7d73f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/959807de1c90480c9c48f0aff3f7d73f 2023-04-27 03:54:49,734 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/959807de1c90480c9c48f0aff3f7d73f, entries=150, sequenceid=248, filesize=11.8 K 2023-04-27 03:54:49,735 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/45a43f0c7d0b48e0ae25cbd999e8ced7 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/45a43f0c7d0b48e0ae25cbd999e8ced7 2023-04-27 03:54:49,738 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/45a43f0c7d0b48e0ae25cbd999e8ced7, entries=150, sequenceid=248, filesize=11.8 K 2023-04-27 03:54:49,739 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for e07ddca27cd19d46379b6939f22f78c7 in 1246ms, sequenceid=248, compaction requested=false 2023-04-27 03:54:49,739 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:49,739 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:49,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:49,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:49,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:49,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:49,741 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,741 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:49,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:49,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:49,741 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:49,741 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:49,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:49,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:49,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:49,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:49,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:49,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5286f102[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:49,742 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:49,743 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:49,744 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:49,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:49,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:49,745 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:49,745 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:49,745 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,745 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:49,745 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:49,746 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:49,746 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:49,746 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,746 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:49,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:49,749 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:49,749 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:49,749 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:49,749 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:49,749 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:49,750 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:49,750 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:49,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:49,750 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:49,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:49,751 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:49,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:49,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:49,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,752 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,754 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,754 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:49,754 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:49,754 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:49,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:49,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:49,754 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:49,754 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:49,755 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:49,755 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:49,755 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:49,755 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:49,755 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:49,755 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:49,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:49,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:49,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:49,816 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e07ddca27cd19d46379b6939f22f78c7 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-27 03:54:49,817 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=A 2023-04-27 03:54:49,817 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:49,817 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=B 2023-04-27 03:54:49,817 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:49,817 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=C 2023-04-27 03:54:49,817 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:49,825 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/f5368c35d39245d98412916eda6146c8 2023-04-27 03:54:49,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:49,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567749830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:49,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567749831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,836 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:49,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:49,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567749833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567749833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,855 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:49,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:49,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:49,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:49,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:49,858 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:49,858 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:49,858 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:49,858 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:49,858 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:49,858 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:49,860 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:49,860 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:49,861 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,861 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:49,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:49,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:49,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:49,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:49,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:49,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:49,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:49,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:49,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:49,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:49,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:49,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:49,862 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:49,862 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:49,862 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:49,862 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:49,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:49,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:49,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:49,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:49,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,864 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:49,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:49,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:49,864 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:49,864 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:49,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:49,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:49,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:49,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:49,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:49,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1471f837[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:49,865 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:49,865 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:49,867 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:49,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:49,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:49,867 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:49,867 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:49,867 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:49,867 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,867 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:49,872 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,872 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:49,872 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:49,872 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:49,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:49,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:49,874 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:49,874 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:49,874 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:49,874 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:49,874 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:49,876 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:49,876 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:49,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:49,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:49,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:49,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:49,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:49,876 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:49,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:49,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:49,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:49,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:49,877 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:49,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:49,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:49,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,878 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:49,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:49,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,880 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:49,880 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:49,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:49,880 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,881 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:49,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:49,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:49,880 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:49,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:49,881 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:49,881 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:49,881 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:49,881 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,881 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:49,881 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:49,881 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:49,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:49,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567749934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:49,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567749934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:49,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567749937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:49,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567749937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,981 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:49,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:49,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:49,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:49,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:49,984 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:49,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:49,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:49,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:49,985 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:49,985 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:49,986 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:49,986 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:49,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:49,987 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,987 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:49,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:49,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:49,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:49,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:49,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:49,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:49,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:49,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:49,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:49,988 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:49,988 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:49,988 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:49,988 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:49,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:49,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:49,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:49,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:49,992 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:49,992 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:49,992 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:49,992 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:49,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:49,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@408631e4[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:49,994 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:49,994 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:49,995 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:49,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:49,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:49,995 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:49,995 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:49,995 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:49,995 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,995 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:49,996 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:49,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,996 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:49,997 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:49,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:49,999 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:49,999 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:49,999 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:49,999 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:49,999 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:50,001 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,001 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,001 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:50,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:50,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:50,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:50,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:50,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:50,001 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,003 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,007 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,007 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,007 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,007 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:50,007 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:50,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:50,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:50,007 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:50,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:50,007 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:50,007 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:50,007 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,007 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:50,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:50,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:50,107 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:50,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:50,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:50,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:50,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:50,110 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:50,110 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:50,110 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:50,110 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:50,111 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,111 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,113 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:50,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:50,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:50,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:50,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:50,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:50,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:50,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:50,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:50,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:50,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:50,115 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:50,115 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:50,115 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:50,115 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:50,115 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:50,115 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:50,115 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:50,115 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:50,115 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,115 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:50,115 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:50,117 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,117 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:50,117 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,117 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:50,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:50,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:50,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:50,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43d2b9cb[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:50,119 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:50,119 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,121 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:50,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:50,121 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,121 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:50,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:50,124 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:50,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,124 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:50,124 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:50,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:50,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:50,126 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:50,126 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:50,126 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:50,126 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:50,126 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:50,129 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,129 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:50,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,129 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:50,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:50,129 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,130 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,133 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,133 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,133 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:50,133 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:50,133 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:50,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:50,133 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:50,133 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,133 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:50,133 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:50,133 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:50,133 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:50,133 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:50,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567750137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:50,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567750138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:50,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567750139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:50,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567750140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,233 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:50,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:50,235 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/c40e7b3ebc2a4f93a4a1797ee0617c61 2023-04-27 03:54:50,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:50,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:50,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:50,237 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:50,237 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:50,237 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:50,237 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:50,237 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,237 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,239 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:50,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:50,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:50,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:50,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:50,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:50,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:50,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:50,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:50,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:50,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:50,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:50,243 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:50,243 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:50,243 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:50,243 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:50,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:50,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:50,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:50,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:50,246 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,246 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:50,247 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,247 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:50,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:50,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26b95e52[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:50,249 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:50,249 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,251 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,251 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:50,251 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:50,251 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:50,251 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,251 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:50,254 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,254 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:50,254 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:50,254 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:50,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:50,256 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:50,256 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:50,256 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:50,256 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:50,256 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:50,262 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,262 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,262 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:50,262 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:50,263 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,266 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,266 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:50,266 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:50,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:50,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:50,266 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:50,266 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,266 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,266 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,266 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,266 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,266 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:50,266 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:50,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:50,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:50,266 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:50,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:50,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:50,367 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:50,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:50,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:50,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:50,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:50,369 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:50,370 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:50,370 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:50,370 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:50,370 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,370 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,372 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:50,372 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:50,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:50,372 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,372 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:50,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:50,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:50,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:50,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:50,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:50,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:50,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:50,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:50,374 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:50,374 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:50,374 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:50,374 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:50,376 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,376 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:50,376 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,376 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:50,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:50,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:50,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d12aaf2[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:50,377 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:50,377 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,380 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:50,380 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,380 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:50,380 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:50,380 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:50,381 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:50,381 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,381 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:50,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:50,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:50,382 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:50,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:50,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:50,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:50,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:50,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:50,384 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:50,384 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:50,384 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:50,384 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:50,384 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:50,385 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,385 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:50,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,385 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:50,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:50,386 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:50,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,387 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:50,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,389 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:50,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:50,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:50,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:50,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:50,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:50,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:50,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:50,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:50,390 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:50,390 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:50,390 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,390 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:50,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:50,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567750440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:50,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567750442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:50,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567750444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:50,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567750444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,490 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:50,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:50,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:50,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:50,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:50,493 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:50,493 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:50,493 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:50,493 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:50,494 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,494 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,496 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:50,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:50,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:50,496 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,497 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,497 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:50,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:50,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:50,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:50,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:50,498 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:50,498 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:50,498 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:50,498 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:50,498 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:50,499 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:50,499 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:50,499 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:50,499 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:50,499 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:50,499 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:50,499 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,499 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:50,499 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:50,500 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,500 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:50,501 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,501 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:50,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:50,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c6ab5a5[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:50,502 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:50,502 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,504 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,504 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:50,504 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:50,504 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:50,504 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,504 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:50,505 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,505 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:50,505 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:50,506 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:50,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:50,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:50,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:50,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:50,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:50,508 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:50,508 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:50,508 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:50,508 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:50,508 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:50,511 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,511 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:50,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,511 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:50,511 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:50,512 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,515 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,515 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:50,515 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,515 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:50,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:50,515 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:50,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:50,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:50,515 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,516 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:50,516 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,516 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:50,516 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:50,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:50,516 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,516 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,616 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:50,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:50,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:50,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:50,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:50,619 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:50,619 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:50,619 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:50,619 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:50,620 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,620 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,621 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:50,621 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:50,622 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,622 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:50,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,622 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:50,623 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:50,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:50,623 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:50,623 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:50,623 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:50,623 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:50,623 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:50,623 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:50,623 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:50,623 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:50,623 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:50,623 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:50,623 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:50,623 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,623 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:50,623 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:50,625 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,625 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,625 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,625 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:50,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:50,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aa04102[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:50,627 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:50,627 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,629 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,629 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,630 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:50,630 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:50,630 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:50,630 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:50,631 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:50,631 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:50,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:50,631 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:50,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:50,633 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:50,634 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:50,634 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:50,634 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:50,634 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:50,635 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,635 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,635 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:50,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:50,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:50,636 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,637 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,643 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:50,643 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:50,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:50,644 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,644 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,644 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,644 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:50,644 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:50,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:50,644 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,644 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:50,644 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,644 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:50,644 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:50,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:50,646 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/5ebdabb24bd542f694bf432fbf59717d 2023-04-27 03:54:50,650 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/f5368c35d39245d98412916eda6146c8 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/f5368c35d39245d98412916eda6146c8 2023-04-27 03:54:50,654 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/f5368c35d39245d98412916eda6146c8, entries=200, sequenceid=267, filesize=14.3 K 2023-04-27 03:54:50,655 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/c40e7b3ebc2a4f93a4a1797ee0617c61 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/c40e7b3ebc2a4f93a4a1797ee0617c61 2023-04-27 03:54:50,658 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/c40e7b3ebc2a4f93a4a1797ee0617c61, entries=150, sequenceid=267, filesize=11.9 K 2023-04-27 03:54:50,659 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/5ebdabb24bd542f694bf432fbf59717d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/5ebdabb24bd542f694bf432fbf59717d 2023-04-27 03:54:50,662 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/5ebdabb24bd542f694bf432fbf59717d, entries=150, sequenceid=267, filesize=11.9 K 2023-04-27 03:54:50,663 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for e07ddca27cd19d46379b6939f22f78c7 in 847ms, sequenceid=267, compaction requested=true 2023-04-27 03:54:50,663 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:50,663 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:50,663 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:50,663 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:50,663 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:50,663 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:50,664 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39210 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:50,664 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/A is initiating minor compaction (all files) 2023-04-27 03:54:50,664 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36770 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:50,664 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/A in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:50,664 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/B is initiating minor compaction (all files) 2023-04-27 03:54:50,664 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/5ac1f73a478c439f95381e70e18a8a2b, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/52587078d9c244c28f2da9695e910002, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/f5368c35d39245d98412916eda6146c8] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=38.3 K 2023-04-27 03:54:50,664 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/B in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:50,665 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/0cbc82a5b4294ba68157c0676bfc02d5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/959807de1c90480c9c48f0aff3f7d73f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/c40e7b3ebc2a4f93a4a1797ee0617c61] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=35.9 K 2023-04-27 03:54:50,665 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 5ac1f73a478c439f95381e70e18a8a2b, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682567687449 2023-04-27 03:54:50,665 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 0cbc82a5b4294ba68157c0676bfc02d5, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682567687449 2023-04-27 03:54:50,665 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 52587078d9c244c28f2da9695e910002, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1682567687576 2023-04-27 03:54:50,665 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 959807de1c90480c9c48f0aff3f7d73f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1682567687576 2023-04-27 03:54:50,665 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting f5368c35d39245d98412916eda6146c8, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682567688698 2023-04-27 03:54:50,666 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting c40e7b3ebc2a4f93a4a1797ee0617c61, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682567688701 2023-04-27 03:54:50,672 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#A#compaction#258 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:50,672 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#B#compaction#259 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:50,681 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/3ce2f9d5686a41c59e7a3a317ab155be as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/3ce2f9d5686a41c59e7a3a317ab155be 2023-04-27 03:54:50,682 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/07f133281b5744e09c2856ff1dd71d7e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/07f133281b5744e09c2856ff1dd71d7e 2023-04-27 03:54:50,686 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/B of e07ddca27cd19d46379b6939f22f78c7 into 3ce2f9d5686a41c59e7a3a317ab155be(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:50,686 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/A of e07ddca27cd19d46379b6939f22f78c7 into 07f133281b5744e09c2856ff1dd71d7e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:50,686 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:50,686 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:50,686 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/B, priority=13, startTime=1682567690663; duration=0sec 2023-04-27 03:54:50,686 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/A, priority=13, startTime=1682567690663; duration=0sec 2023-04-27 03:54:50,686 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:50,686 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:50,686 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:50,688 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36736 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:50,688 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/C is initiating minor compaction (all files) 2023-04-27 03:54:50,688 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/C in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:50,688 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/5c8836737685436f8354736f3911418a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/45a43f0c7d0b48e0ae25cbd999e8ced7, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/5ebdabb24bd542f694bf432fbf59717d] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=35.9 K 2023-04-27 03:54:50,688 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 5c8836737685436f8354736f3911418a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682567687449 2023-04-27 03:54:50,688 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 45a43f0c7d0b48e0ae25cbd999e8ced7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1682567687576 2023-04-27 03:54:50,689 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 5ebdabb24bd542f694bf432fbf59717d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682567688701 2023-04-27 03:54:50,693 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#C#compaction#260 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:50,744 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:50,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:50,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:50,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:50,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:50,747 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:50,747 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:50,747 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:50,747 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:50,748 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,748 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,749 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:50,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:50,749 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,749 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:50,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:50,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:50,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:50,751 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:50,751 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:50,751 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:50,751 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:50,751 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:50,751 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:50,751 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:50,751 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e07ddca27cd19d46379b6939f22f78c7 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-27 03:54:50,752 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=A 2023-04-27 03:54:50,752 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:50,752 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=B 2023-04-27 03:54:50,752 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:50,752 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=C 2023-04-27 03:54:50,752 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:50,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:50,943 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:50,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:50,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567750955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:50,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567750958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:50,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:50,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567750958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:50,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567750959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:51,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567751059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:51,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567751062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:51,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567751063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:51,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567751063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,102 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/23b2500bfcbe420897251ccd50fed5d5 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/23b2500bfcbe420897251ccd50fed5d5 2023-04-27 03:54:51,111 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/C of e07ddca27cd19d46379b6939f22f78c7 into 23b2500bfcbe420897251ccd50fed5d5(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:51,111 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:51,111 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/C, priority=13, startTime=1682567690663; duration=0sec 2023-04-27 03:54:51,111 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:51,160 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/44b8721facbc40c1a59e8f1a2c4f9ba2 2023-04-27 03:54:51,174 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/980b6e3ee61742ea9bebf199322ec608 2023-04-27 03:54:51,189 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/13b91643d0234cb784596e4cecf403d0 2023-04-27 03:54:51,195 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/44b8721facbc40c1a59e8f1a2c4f9ba2 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/44b8721facbc40c1a59e8f1a2c4f9ba2 2023-04-27 03:54:51,199 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/44b8721facbc40c1a59e8f1a2c4f9ba2, entries=150, sequenceid=285, filesize=11.9 K 2023-04-27 03:54:51,199 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/980b6e3ee61742ea9bebf199322ec608 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/980b6e3ee61742ea9bebf199322ec608 2023-04-27 03:54:51,204 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/980b6e3ee61742ea9bebf199322ec608, entries=150, sequenceid=285, filesize=11.9 K 2023-04-27 03:54:51,204 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/13b91643d0234cb784596e4cecf403d0 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/13b91643d0234cb784596e4cecf403d0 2023-04-27 03:54:51,217 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/13b91643d0234cb784596e4cecf403d0, entries=150, sequenceid=285, filesize=11.9 K 2023-04-27 03:54:51,218 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for e07ddca27cd19d46379b6939f22f78c7 in 467ms, sequenceid=285, compaction requested=false 2023-04-27 03:54:51,218 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:51,218 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:51,218 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:51,218 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:51,219 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,220 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:51,220 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:51,221 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,222 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:51,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:51,222 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,222 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:51,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:51,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:51,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:51,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:51,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7607d7ca[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:51,224 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:51,224 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,225 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,225 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:51,225 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,226 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:51,226 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:51,226 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:51,229 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,229 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:51,229 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:51,229 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:51,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:51,234 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:51,234 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:51,234 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:51,234 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:51,234 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:51,236 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,236 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:51,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,236 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:51,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:51,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:51,237 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:51,238 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:51,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,240 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,241 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,241 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,241 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,241 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:51,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:51,241 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:51,241 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:51,241 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,241 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:51,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:51,241 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:51,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:51,241 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:51,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:51,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:51,264 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e07ddca27cd19d46379b6939f22f78c7 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-27 03:54:51,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=A 2023-04-27 03:54:51,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:51,265 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=B 2023-04-27 03:54:51,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:51,265 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=C 2023-04-27 03:54:51,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:51,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:51,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567751276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:51,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567751276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:51,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567751277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:51,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567751277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,341 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:51,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:51,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:51,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:51,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:51,345 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:51,345 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:51,345 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:51,345 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:51,345 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,345 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,347 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,347 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:51,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:51,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:51,347 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,347 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:51,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:51,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:51,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:51,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:51,349 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:51,349 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:51,349 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:51,349 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:51,349 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:51,349 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:51,349 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:51,349 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:51,349 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:51,349 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:51,349 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:51,349 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,349 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:51,349 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,351 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:51,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:51,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:51,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:51,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5eeaa6c2[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:51,353 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:51,353 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,355 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,355 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:51,356 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:51,356 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:51,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:51,357 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:51,357 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:51,357 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,357 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:51,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:51,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:51,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:51,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:51,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:51,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:51,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:51,360 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:51,360 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:51,360 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:51,360 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:51,360 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:51,362 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,362 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:51,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:51,362 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:51,362 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:51,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:51,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:51,364 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,367 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,367 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:51,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:51,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:51,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:51,367 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:51,367 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:51,367 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:51,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:51,367 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,367 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:51,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:51,367 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,368 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,368 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,368 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,368 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:51,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567751380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:51,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37824 deadline: 1682567751380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:51,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567751380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:51,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567751380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,467 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:51,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:51,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:51,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:51,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:51,471 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:51,471 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:51,471 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:51,471 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:51,471 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,471 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,473 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:51,473 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:51,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:51,474 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,474 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:51,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:51,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:51,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:51,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:51,475 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:51,475 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:51,475 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:51,475 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:51,475 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:51,475 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:51,475 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:51,475 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:51,475 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:51,475 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:51,475 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:51,475 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,475 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:51,475 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:51,477 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,477 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:51,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:51,477 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,477 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:51,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:51,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:51,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:51,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:51,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4734e727[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:51,479 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:51,479 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,480 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:51,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:51,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:51,480 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,480 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:51,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:51,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:51,481 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:51,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:51,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:51,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:51,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:51,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:51,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:51,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:51,484 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:51,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:51,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:51,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:51,484 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:51,486 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,486 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:51,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,486 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:51,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:51,487 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:51,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,488 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:51,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:51,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:51,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:51,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:51,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,497 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:51,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:51,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:51,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:51,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:51,498 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:51,498 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:51,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:51,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567751583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:51,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37824 deadline: 1682567751583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:51,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567751585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:51,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567751585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,598 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:51,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:51,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:51,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:51,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:51,601 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:51,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:51,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:51,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:51,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,603 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,603 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:51,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:51,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:51,603 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,603 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:51,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:51,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:51,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:51,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:51,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:51,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:51,605 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:51,605 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:51,605 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:51,605 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:51,605 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:51,605 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:51,605 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:51,605 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:51,605 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:51,605 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,605 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:51,605 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:51,606 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,607 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:51,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:51,607 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,607 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:51,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:51,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:51,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:51,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:51,608 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:51,608 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f040692[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:51,610 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,611 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:51,611 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:51,611 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:51,612 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:51,612 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:51,612 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:51,613 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,613 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:51,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:51,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:51,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:51,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:51,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:51,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:51,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:51,615 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:51,616 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:51,616 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:51,616 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:51,616 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:51,617 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,617 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:51,618 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:51,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:51,618 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:51,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,619 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:51,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,623 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:51,623 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:51,623 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:51,623 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,623 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:51,623 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:51,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:51,623 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,623 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,623 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,623 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:51,623 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:51,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:51,623 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:51,672 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/8845cb603fd148a5b3efc527d8544388 2023-04-27 03:54:51,724 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:51,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:51,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:51,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:51,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:51,727 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:51,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:51,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:51,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:51,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,729 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:51,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:51,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:51,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,730 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:51,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:51,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:51,731 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:51,731 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:51,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:51,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:51,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:51,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:51,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:51,731 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:51,731 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:51,731 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:51,731 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:51,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:51,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:51,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:51,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:51,733 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,733 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:51,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:51,733 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,734 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:51,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:51,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:51,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:51,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:51,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17f5333f[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:51,735 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:51,735 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,736 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,736 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:51,736 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:51,736 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:51,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:51,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:51,738 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:51,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:51,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:51,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:51,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:51,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:51,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:51,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:51,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:51,740 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:51,740 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:51,740 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:51,740 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:51,740 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:51,743 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,743 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:51,743 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,744 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:51,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,746 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:51,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,748 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:51,748 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:51,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:51,748 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:51,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:51,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:51,748 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:51,748 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:51,748 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,748 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:51,748 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:51,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:51,748 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,848 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:51,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:51,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:51,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:51,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:51,852 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:51,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:51,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:51,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:51,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,853 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:51,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:51,854 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,854 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,854 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:51,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:51,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:51,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:51,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:51,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:51,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:51,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:51,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:51,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:51,855 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:51,855 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:51,855 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:51,855 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:51,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:51,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:51,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:51,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:51,857 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,857 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,857 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:51,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:51,858 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:51,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:51,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:51,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:51,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:51,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ca25bb5[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:51,859 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:51,859 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,862 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:51,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:51,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:51,862 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,862 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:51,863 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,863 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:51,863 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:51,864 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:51,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:51,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:51,866 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:51,866 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:51,866 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:51,866 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:51,866 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:51,867 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,867 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:51,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:51,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,867 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:51,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:51,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:51,868 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:51,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:51,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,869 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:51,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,871 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,871 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:51,871 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:51,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:51,871 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,871 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:51,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:51,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:51,871 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:51,871 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,872 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:51,872 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:51,872 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,872 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:51,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:51,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:51,872 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:51,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37824 deadline: 1682567751887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:51,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567751887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:51,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567751888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:51,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567751890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,972 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:51,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:51,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:51,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:51,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:51,975 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:51,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:51,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:51,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:51,976 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,976 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,978 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,978 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:51,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:51,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:51,978 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,979 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:51,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:51,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:51,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:51,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:51,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:51,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:51,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:51,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:51,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:51,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:51,980 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:51,980 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:51,980 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:51,980 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:51,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:51,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:51,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:51,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:51,985 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:51,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:51,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:51,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:51,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:51,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:51,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:51,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ccf9ad9[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:51,987 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:51,987 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,988 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:51,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:51,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:51,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:51,988 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,988 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:51,989 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:51,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:51,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:51,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:51,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:51,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:51,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:51,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:51,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:51,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:51,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:51,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:51,992 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:51,992 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:51,992 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:51,992 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:51,992 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:51,999 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,999 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,999 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:51,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:51,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:52,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:52,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:52,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:52,002 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:52,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:52,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,003 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,005 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:52,005 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,005 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:52,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:52,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:52,005 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:52,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:52,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:52,005 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,006 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:52,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,006 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:52,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:52,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:52,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:52,094 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/6fd3ee6c5fc0444d949ee036415acb9b 2023-04-27 03:54:52,106 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:52,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:52,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:52,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:52,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:52,122 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:52,122 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:52,122 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:52,122 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:52,123 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,123 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:52,124 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:52,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:52,124 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,124 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:52,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:52,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:52,135 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:52,135 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:52,135 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:52,135 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:52,135 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:52,142 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:52,142 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:52,143 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:52,143 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:52,143 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:52,143 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:52,143 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:52,143 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:52,143 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,143 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:52,143 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:52,145 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,145 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:52,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:52,145 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,145 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:52,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:52,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:52,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3802344[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:52,147 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:52,147 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,149 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,149 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:52,149 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:52,149 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:52,149 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,149 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:52,150 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:52,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:52,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:52,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:52,153 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:52,153 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:52,153 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:52,153 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:52,153 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:52,155 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,155 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,155 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:52,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:52,156 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,157 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,159 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:52,159 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:52,159 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:52,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:52,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:52,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:52,159 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:52,159 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,160 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,160 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,160 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,160 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,160 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:52,160 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:52,160 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:52,260 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:52,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:52,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:52,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:52,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:52,263 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:52,263 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:52,263 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:52,263 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:52,264 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,264 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,266 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,266 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:52,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:52,266 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,267 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:52,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:52,267 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:52,267 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:52,267 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:52,267 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:52,267 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:52,268 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:52,268 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:52,268 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:52,268 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:52,268 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:52,268 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:52,268 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:52,268 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:52,268 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,268 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:52,268 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:52,269 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,269 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:52,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:52,270 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,270 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:52,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:52,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:52,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:52,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:52,271 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:52,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d5294c0[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:52,271 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,272 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,272 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:52,272 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:52,272 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:52,272 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,272 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:52,274 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,274 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:52,274 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:52,274 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:52,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:52,276 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:52,276 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:52,276 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:52,276 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:52,276 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:52,279 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,279 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:52,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,279 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:52,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:52,279 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,280 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:52,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,282 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,283 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,283 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:52,283 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:52,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:52,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:52,283 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,283 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:52,283 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:52,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:52,283 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:52,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:52,283 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,283 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,283 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:52,283 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:52,383 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:52,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:52,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:52,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:52,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:52,387 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:52,387 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:52,387 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:52,387 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:52,387 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,387 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,388 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,388 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:52,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:52,389 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,389 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:52,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:52,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:52,390 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:52,390 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:52,390 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:52,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:52,390 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:52,390 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:52,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37824 deadline: 1682567752389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,390 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:52,390 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:52,391 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:52,391 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:52,391 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:52,391 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:52,391 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,391 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:52,391 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:52,392 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,392 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:52,393 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,393 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:52,393 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:52,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567752391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:52,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:52,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:52,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567752393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33a31f2c[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:52,394 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:52,394 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:52,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567752394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,400 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,400 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:52,400 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:52,400 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,400 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:52,400 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:52,402 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,402 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:52,402 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:52,402 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:52,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:52,404 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:52,404 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:52,404 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:52,404 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:52,404 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:52,406 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,406 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,406 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:52,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:52,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:52,407 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:52,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:52,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:52,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:52,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,408 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:52,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,411 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,411 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,411 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,411 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,411 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,411 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,411 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:52,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:52,411 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:52,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:52,412 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:52,412 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:52,412 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:52,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:52,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:52,412 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:52,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:52,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:52,512 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:52,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:52,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:52,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:52,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:52,515 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:52,515 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:52,516 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:52,516 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:52,516 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,516 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,517 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,517 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:52,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:52,518 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,518 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:52,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:52,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:52,521 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:52,521 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:52,521 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:52,521 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:52,523 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,523 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:52,523 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,523 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:52,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:52,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@333a42d3[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:52,525 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:52,525 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,526 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,526 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,526 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:52,526 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:52,526 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:52,526 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:52,528 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,528 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:52,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:52,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:52,528 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:52,528 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:52,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:52,531 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:52,531 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:52,531 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:52,531 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:52,531 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:52,532 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:52,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:52,533 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:52,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,533 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:52,533 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:52,534 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:52,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:52,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:52,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,536 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:52,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:52,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:52,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:52,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:52,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:52,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:52,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,536 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:52,537 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:52,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:52,537 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:52,539 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/f6e7b6803d9345a0b5200b922d16fa33 2023-04-27 03:54:52,545 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/8845cb603fd148a5b3efc527d8544388 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/8845cb603fd148a5b3efc527d8544388 2023-04-27 03:54:52,549 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/8845cb603fd148a5b3efc527d8544388, entries=150, sequenceid=304, filesize=11.9 K 2023-04-27 03:54:52,549 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/6fd3ee6c5fc0444d949ee036415acb9b as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/6fd3ee6c5fc0444d949ee036415acb9b 2023-04-27 03:54:52,553 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/6fd3ee6c5fc0444d949ee036415acb9b, entries=150, sequenceid=304, filesize=11.9 K 2023-04-27 03:54:52,553 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/f6e7b6803d9345a0b5200b922d16fa33 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/f6e7b6803d9345a0b5200b922d16fa33 2023-04-27 03:54:52,557 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/f6e7b6803d9345a0b5200b922d16fa33, entries=150, sequenceid=304, filesize=11.9 K 2023-04-27 03:54:52,557 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for e07ddca27cd19d46379b6939f22f78c7 in 1293ms, sequenceid=304, compaction requested=true 2023-04-27 03:54:52,557 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:52,558 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:52,558 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:52,558 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:52,558 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:52,558 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:52,558 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:52,559 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:52,559 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/A is initiating minor compaction (all files) 2023-04-27 03:54:52,559 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/B is initiating minor compaction (all files) 2023-04-27 03:54:52,559 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/A in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:52,559 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/B in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:52,559 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/07f133281b5744e09c2856ff1dd71d7e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/44b8721facbc40c1a59e8f1a2c4f9ba2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/8845cb603fd148a5b3efc527d8544388] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=36.3 K 2023-04-27 03:54:52,559 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/3ce2f9d5686a41c59e7a3a317ab155be, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/980b6e3ee61742ea9bebf199322ec608, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/6fd3ee6c5fc0444d949ee036415acb9b] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=36.3 K 2023-04-27 03:54:52,559 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 3ce2f9d5686a41c59e7a3a317ab155be, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682567688701 2023-04-27 03:54:52,559 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 07f133281b5744e09c2856ff1dd71d7e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682567688701 2023-04-27 03:54:52,560 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 980b6e3ee61742ea9bebf199322ec608, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682567689826 2023-04-27 03:54:52,560 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 44b8721facbc40c1a59e8f1a2c4f9ba2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682567689826 2023-04-27 03:54:52,560 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 6fd3ee6c5fc0444d949ee036415acb9b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682567690950 2023-04-27 03:54:52,560 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 8845cb603fd148a5b3efc527d8544388, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682567690950 2023-04-27 03:54:52,567 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#B#compaction#267 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:52,567 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#A#compaction#268 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:52,577 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/1ed6d0622217417385d7d348279ca21d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/1ed6d0622217417385d7d348279ca21d 2023-04-27 03:54:52,580 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/B of e07ddca27cd19d46379b6939f22f78c7 into 1ed6d0622217417385d7d348279ca21d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:52,580 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:52,580 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/B, priority=13, startTime=1682567692558; duration=0sec 2023-04-27 03:54:52,581 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:52,581 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:52,581 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:52,582 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/C is initiating minor compaction (all files) 2023-04-27 03:54:52,582 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/C in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:52,582 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/23b2500bfcbe420897251ccd50fed5d5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/13b91643d0234cb784596e4cecf403d0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/f6e7b6803d9345a0b5200b922d16fa33] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=36.3 K 2023-04-27 03:54:52,582 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 23b2500bfcbe420897251ccd50fed5d5, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682567688701 2023-04-27 03:54:52,582 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 13b91643d0234cb784596e4cecf403d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682567689826 2023-04-27 03:54:52,583 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting f6e7b6803d9345a0b5200b922d16fa33, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682567690950 2023-04-27 03:54:52,587 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#C#compaction#269 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:52,594 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/98bfbad09926456386e74811381c415e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/98bfbad09926456386e74811381c415e 2023-04-27 03:54:52,598 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/C of e07ddca27cd19d46379b6939f22f78c7 into 98bfbad09926456386e74811381c415e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:52,598 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:52,598 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/C, priority=13, startTime=1682567692558; duration=0sec 2023-04-27 03:54:52,598 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:52,637 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:52,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:52,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:52,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:52,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:52,640 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:52,640 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:52,640 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:52,640 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:52,640 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,640 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,642 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,642 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:52,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,642 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:52,642 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:52,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:52,643 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:52,643 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:52,643 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:52,643 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:52,643 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:52,643 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:52,644 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:52,644 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:52,644 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:52,644 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:52,644 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e07ddca27cd19d46379b6939f22f78c7 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-27 03:54:52,644 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=A 2023-04-27 03:54:52,644 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:52,644 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=B 2023-04-27 03:54:52,644 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:52,644 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=C 2023-04-27 03:54:52,644 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:52,652 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/d124f23070bc430abbb6fa68c91a761a 2023-04-27 03:54:52,662 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/d64f78e5d190496d9746b27e59e7a50b 2023-04-27 03:54:52,978 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/d4c8ca7859e64ae1a92e6d22abcd5f5a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/d4c8ca7859e64ae1a92e6d22abcd5f5a 2023-04-27 03:54:52,983 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/A of e07ddca27cd19d46379b6939f22f78c7 into d4c8ca7859e64ae1a92e6d22abcd5f5a(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:52,983 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:52,983 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/A, priority=13, startTime=1682567692558; duration=0sec 2023-04-27 03:54:52,983 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:53,072 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/0c9cd4f3e7904f39ba10f59fb1cc6605 2023-04-27 03:54:53,077 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/d124f23070bc430abbb6fa68c91a761a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/d124f23070bc430abbb6fa68c91a761a 2023-04-27 03:54:53,082 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/d124f23070bc430abbb6fa68c91a761a, entries=150, sequenceid=321, filesize=11.9 K 2023-04-27 03:54:53,082 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/d64f78e5d190496d9746b27e59e7a50b as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/d64f78e5d190496d9746b27e59e7a50b 2023-04-27 03:54:53,086 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/d64f78e5d190496d9746b27e59e7a50b, entries=150, sequenceid=321, filesize=11.9 K 2023-04-27 03:54:53,087 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/0c9cd4f3e7904f39ba10f59fb1cc6605 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/0c9cd4f3e7904f39ba10f59fb1cc6605 2023-04-27 03:54:53,090 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/0c9cd4f3e7904f39ba10f59fb1cc6605, entries=150, sequenceid=321, filesize=11.9 K 2023-04-27 03:54:53,090 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for e07ddca27cd19d46379b6939f22f78c7 in 446ms, sequenceid=321, compaction requested=false 2023-04-27 03:54:53,090 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:53,090 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:53,091 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:53,091 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:53,091 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,091 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:53,091 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:53,093 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,093 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:53,093 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,094 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:53,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:53,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3122ddf3[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:53,095 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:53,095 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,096 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,096 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:53,097 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:53,097 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:53,097 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,098 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:53,099 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:53,099 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,099 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:53,099 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:53,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:53,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:53,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:53,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:53,101 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:53,102 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:53,102 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:53,102 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:53,102 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:53,104 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,104 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:53,104 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:53,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,105 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:53,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,106 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:53,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:53,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:53,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:53,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,110 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:53,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:53,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:53,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:53,111 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,111 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:53,111 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:53,111 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,111 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,111 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,111 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,211 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:53,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:53,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:53,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:53,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:53,214 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:53,214 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:53,214 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:53,214 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:53,215 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,215 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,216 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,216 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:53,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:53,216 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,216 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:53,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:53,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:53,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:53,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:53,218 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:53,218 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:53,218 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:53,218 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:53,218 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:53,218 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:53,218 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:53,218 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:53,218 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:53,218 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:53,218 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:53,218 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,218 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:53,219 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:53,220 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,220 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:53,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:53,221 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,221 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:53,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:53,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@187fab83[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:53,222 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:53,222 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,223 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,224 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:53,224 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:53,224 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:53,224 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,224 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:53,226 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:53,226 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,226 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:53,226 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:53,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:53,229 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:53,229 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:53,229 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:53,229 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:53,229 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:53,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:53,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:53,233 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:53,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,235 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,237 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,237 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,237 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:53,237 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:53,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:53,237 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,237 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,237 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:53,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:53,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:53,238 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:53,237 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:53,238 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:53,237 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:53,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:53,238 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,338 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:53,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:53,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:53,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:53,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:53,341 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:53,341 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:53,341 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:53,341 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:53,342 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,342 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,343 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,343 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:53,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:53,343 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,344 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:53,345 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:53,345 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:53,345 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:53,345 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:53,345 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:53,345 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:53,345 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:53,345 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:53,345 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:53,345 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:53,345 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:53,345 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:53,345 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:53,345 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,345 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:53,345 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:53,348 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:53,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:53,350 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:53,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@338e8ddf[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:53,350 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,351 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,351 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:53,352 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,352 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:53,353 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:53,353 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:53,353 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,353 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:53,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:53,356 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:53,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:53,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:53,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:53,356 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:53,358 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,358 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:53,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,358 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:53,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:53,359 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,360 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,364 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,364 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,364 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:53,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:53,364 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:53,364 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:53,364 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:53,364 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:53,364 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:53,364 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:53,364 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,365 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,365 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:53,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:53,405 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e07ddca27cd19d46379b6939f22f78c7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:54:53,405 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=A 2023-04-27 03:54:53,405 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:53,405 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=B 2023-04-27 03:54:53,405 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:53,405 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=C 2023-04-27 03:54:53,405 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:53,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:53,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37824 deadline: 1682567753442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:53,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567753442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567693418, max timestamp=1682567693445Num uniques 859; 2023-04-27 03:54:53,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:54:53,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567693418, max timestamp=1682567693445Num uniques 859; 2023-04-27 03:54:53,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:54:53,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567693418, max timestamp=1682567693445Num uniques 859; 2023-04-27 03:54:53,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:54:53,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:53,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567753443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,464 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:53,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:53,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:53,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:53,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:53,467 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:53,467 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:53,467 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:53,468 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:53,468 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,468 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,470 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:53,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:53,470 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,470 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,471 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:53,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:53,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:53,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:53,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:53,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:53,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:53,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:53,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:53,472 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:53,472 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:53,472 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:53,472 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:53,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:53,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:53,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:53,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:53,475 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,475 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:53,475 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,475 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:53,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:53,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52436da7[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:53,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:53,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,479 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,479 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:53,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:53,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:53,480 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,480 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:53,482 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:53,482 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:53,482 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,482 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:53,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:53,485 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:53,485 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:53,485 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:53,485 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:53,485 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:53,489 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,489 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:53,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,489 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:53,489 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:53,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:53,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:53,490 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,494 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:53,494 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:53,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:53,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:53,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:53,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:53,494 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:53,494 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,495 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:53,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:53,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:53,495 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,495 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:53,495 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:53,495 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,495 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,495 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,495 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:53,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567753499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:53,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37824 deadline: 1682567753547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:53,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567753547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:53,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567753549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,595 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:53,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:53,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:53,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:53,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:53,598 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:53,598 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:53,598 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:53,598 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:53,598 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,599 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,600 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,600 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:53,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:53,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:53,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:53,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:53,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:53,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:53,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:53,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:53,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:53,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:53,602 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:53,602 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:53,602 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:53,602 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:53,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:53,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:53,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:53,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:53,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:53,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567753602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,604 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:53,605 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,605 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:53,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:53,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7529ef59[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:53,606 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:53,606 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,608 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,608 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:53,608 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:53,608 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:53,608 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,609 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:53,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:53,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:53,610 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:53,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:53,612 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:53,612 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:53,612 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:53,612 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:53,612 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:53,614 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,614 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:53,614 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:53,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:53,615 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:53,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:53,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:53,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,616 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:53,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,627 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,627 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:53,627 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,627 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:53,627 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:53,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:53,627 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:53,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:53,627 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:53,627 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:53,627 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:53,627 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:53,627 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,627 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:53,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:53,727 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:53,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:53,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:53,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:53,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:53,734 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:53,734 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:53,734 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:53,734 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:53,734 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,734 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,736 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:53,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:53,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:53,737 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:53,737 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:53,737 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:53,737 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:53,737 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:53,737 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:53,737 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:53,737 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:53,737 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:53,737 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:53,737 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:53,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,739 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:53,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:53,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:53,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10ef61cc[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:53,741 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:53,741 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,743 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,743 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:53,743 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,743 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:53,743 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:53,743 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:53,745 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,745 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:53,745 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:53,745 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:53,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:53,747 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:53,747 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:53,747 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:53,747 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:53,747 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:53,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:53,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:53,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:53,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:53,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:53,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:53,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,749 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,751 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:53,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37824 deadline: 1682567753749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:53,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567753750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,753 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:53,753 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:53,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:53,753 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:53,754 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:53,754 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:53,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:53,753 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:53,753 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,754 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:53,754 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,754 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,754 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,754 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:53,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:53,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567753752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:53,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567753804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,813 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/ccd26d3eb8a548baa245614f4027e55f 2023-04-27 03:54:53,836 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/d341ddfb26ba4984b95f89be1d511566 2023-04-27 03:54:53,854 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:53,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:53,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:53,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:53,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:53,857 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:53,857 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:53,857 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:53,857 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:53,857 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,858 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,860 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:53,860 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:53,860 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,861 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:53,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:53,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:53,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:53,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:53,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:53,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:53,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:53,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:53,862 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:53,862 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:53,862 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:53,862 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:53,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:53,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:53,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:53,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:53,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,864 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:53,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:53,864 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,864 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:53,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:53,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:53,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57c8f67b[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:53,866 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:53,866 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,867 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,867 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:53,867 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,867 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:53,867 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:53,868 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:53,869 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:53,869 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,869 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:53,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:53,869 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:53,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:53,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:53,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:53,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:53,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:53,873 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:53,873 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:53,873 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:53,873 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:53,873 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:53,876 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,876 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:53,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:53,876 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:53,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:53,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:53,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:53,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,877 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:53,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:53,878 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,880 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,880 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:53,880 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:53,880 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:53,880 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:53,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:53,880 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:53,880 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:53,881 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,881 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,881 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:53,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:53,881 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:53,881 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:53,981 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:53,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:53,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:53,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:53,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:53,985 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:53,985 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:53,985 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:53,985 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:53,985 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,985 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,986 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,986 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:53,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:53,987 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,987 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:53,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:53,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:53,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:53,988 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:53,988 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:53,988 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:53,988 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:53,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:53,990 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,990 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:53,991 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,991 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:53,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:53,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eaf9770[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:53,993 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:53,993 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,995 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:53,995 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:53,995 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:53,995 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:53,995 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,995 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:53,997 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,997 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,997 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:53,997 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:54,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:54,000 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:54,000 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:54,000 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:54,000 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:54,000 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:54,001 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:54,001 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,001 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:54,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,002 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:54,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:54,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:54,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:54,003 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,008 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,008 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:54,008 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:54,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:54,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:54,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:54,008 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:54,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:54,008 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:54,008 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:54,009 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:54,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:54,009 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,009 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:54,009 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:54,009 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:54,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:54,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:54,009 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:54,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37824 deadline: 1682567754053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:54,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567754053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:54,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567754057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,109 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:54,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:54,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:54,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567754108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:54,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:54,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:54,112 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:54,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:54,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:54,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:54,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:54,118 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:54,118 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:54,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:54,118 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,118 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:54,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:54,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:54,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:54,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:54,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:54,119 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:54,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:54,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:54,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:54,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:54,120 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:54,120 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:54,120 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:54,120 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:54,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:54,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:54,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:54,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:54,122 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:54,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:54,123 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:54,123 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:54,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:54,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2446f0db[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:54,124 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:54,124 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:54,125 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:54,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:54,126 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:54,126 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:54,126 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,126 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:54,126 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:54,128 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,128 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:54,128 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:54,128 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:54,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:54,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:54,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:54,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:54,130 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:54,130 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:54,130 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:54,130 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:54,130 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:54,132 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,132 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:54,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:54,132 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:54,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:54,133 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:54,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:54,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,134 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,136 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:54,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:54,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:54,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:54,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:54,137 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:54,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:54,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:54,137 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:54,137 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:54,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:54,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:54,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:54,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:54,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:54,237 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:54,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:54,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:54,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:54,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:54,241 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:54,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:54,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:54,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:54,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:54,242 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,242 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:54,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:54,243 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,243 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:54,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:54,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:54,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:54,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:54,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:54,245 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:54,245 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:54,245 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:54,245 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:54,245 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:54,245 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:54,245 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,245 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:54,245 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:54,246 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/f6fb8587922e4dc49aef71e3e284fffe 2023-04-27 03:54:54,248 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:54,248 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:54,248 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:54,248 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:54,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:54,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fcc149e[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:54,250 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:54,250 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:54,251 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/ccd26d3eb8a548baa245614f4027e55f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/ccd26d3eb8a548baa245614f4027e55f 2023-04-27 03:54:54,251 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:54,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:54,252 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:54,252 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:54,252 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:54,252 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,252 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:54,253 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:54,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:54,253 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:54,253 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:54,253 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:54,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:54,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:54,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,255 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/ccd26d3eb8a548baa245614f4027e55f, entries=200, sequenceid=335, filesize=14.3 K 2023-04-27 03:54:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:54,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:54,256 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/d341ddfb26ba4984b95f89be1d511566 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/d341ddfb26ba4984b95f89be1d511566 2023-04-27 03:54:54,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:54,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:54,256 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:54,256 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:54,257 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:54,257 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:54,257 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:54,259 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,259 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:54,259 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:54,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,259 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/d341ddfb26ba4984b95f89be1d511566, entries=150, sequenceid=335, filesize=11.9 K 2023-04-27 03:54:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:54,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:54,259 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:54,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,260 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/f6fb8587922e4dc49aef71e3e284fffe as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/f6fb8587922e4dc49aef71e3e284fffe 2023-04-27 03:54:54,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:54,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:54,261 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:54,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:54,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,263 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/f6fb8587922e4dc49aef71e3e284fffe, entries=150, sequenceid=335, filesize=11.9 K 2023-04-27 03:54:54,264 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e07ddca27cd19d46379b6939f22f78c7 in 859ms, sequenceid=335, compaction requested=true 2023-04-27 03:54:54,264 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:54,264 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:54,264 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:54,264 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:54,264 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:54,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:54,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:54,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:54,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:54,264 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:54,264 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:54,264 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:54,264 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:54,265 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:54,265 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:54,265 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:54,264 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:54,265 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,265 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:54,265 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:54,265 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,265 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39714 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:54,265 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:54,265 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/A is initiating minor compaction (all files) 2023-04-27 03:54:54,266 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:54,266 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/A in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:54,266 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/B is initiating minor compaction (all files) 2023-04-27 03:54:54,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:54,266 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/B in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:54,266 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/d4c8ca7859e64ae1a92e6d22abcd5f5a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/d124f23070bc430abbb6fa68c91a761a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/ccd26d3eb8a548baa245614f4027e55f] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=38.8 K 2023-04-27 03:54:54,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:54,266 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/1ed6d0622217417385d7d348279ca21d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/d64f78e5d190496d9746b27e59e7a50b, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/d341ddfb26ba4984b95f89be1d511566] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=36.4 K 2023-04-27 03:54:54,266 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting d4c8ca7859e64ae1a92e6d22abcd5f5a, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682567690950 2023-04-27 03:54:54,266 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 1ed6d0622217417385d7d348279ca21d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682567690950 2023-04-27 03:54:54,267 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting d124f23070bc430abbb6fa68c91a761a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1682567691275 2023-04-27 03:54:54,267 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting d64f78e5d190496d9746b27e59e7a50b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1682567691275 2023-04-27 03:54:54,267 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting ccd26d3eb8a548baa245614f4027e55f, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1682567693402 2023-04-27 03:54:54,267 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting d341ddfb26ba4984b95f89be1d511566, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1682567693402 2023-04-27 03:54:54,272 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#A#compaction#276 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:54,273 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#B#compaction#277 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:54,282 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/61934ef7a6d8418abcf8947210fc8bee as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/61934ef7a6d8418abcf8947210fc8bee 2023-04-27 03:54:54,286 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/B of e07ddca27cd19d46379b6939f22f78c7 into 61934ef7a6d8418abcf8947210fc8bee(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:54,286 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:54,286 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/B, priority=13, startTime=1682567694265; duration=0sec 2023-04-27 03:54:54,286 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:54,286 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:54,287 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:54,287 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/C is initiating minor compaction (all files) 2023-04-27 03:54:54,287 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/C in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:54,287 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/98bfbad09926456386e74811381c415e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/0c9cd4f3e7904f39ba10f59fb1cc6605, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/f6fb8587922e4dc49aef71e3e284fffe] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=36.4 K 2023-04-27 03:54:54,287 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 98bfbad09926456386e74811381c415e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682567690950 2023-04-27 03:54:54,288 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 0c9cd4f3e7904f39ba10f59fb1cc6605, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1682567691275 2023-04-27 03:54:54,288 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting f6fb8587922e4dc49aef71e3e284fffe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1682567693402 2023-04-27 03:54:54,295 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#C#compaction#278 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:54,310 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/04a66e0639e14d329a583d5187d6b41d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/04a66e0639e14d329a583d5187d6b41d 2023-04-27 03:54:54,313 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/C of e07ddca27cd19d46379b6939f22f78c7 into 04a66e0639e14d329a583d5187d6b41d(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:54,313 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:54,313 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/C, priority=13, startTime=1682567694265; duration=0sec 2023-04-27 03:54:54,314 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:54,365 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:54,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:54,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:54,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:54,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:54,368 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:54,368 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:54,368 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:54,368 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:54,369 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,369 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:54,370 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:54,370 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:54,370 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,371 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:54,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:54,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:54,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:54,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:54,372 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:54,372 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:54,372 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:54,372 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:54,372 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:54,372 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:54,372 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:54,372 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e07ddca27cd19d46379b6939f22f78c7 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-27 03:54:54,372 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=A 2023-04-27 03:54:54,372 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:54,372 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=B 2023-04-27 03:54:54,373 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:54,373 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=C 2023-04-27 03:54:54,373 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:54,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:54,560 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:54,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:54,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37824 deadline: 1682567754568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:54,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567754568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:54,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567754571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:54,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567754613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:54,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37824 deadline: 1682567754671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:54,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567754672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:54,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567754673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,681 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/c63f23ca16d24b53a51e051021398738 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/c63f23ca16d24b53a51e051021398738 2023-04-27 03:54:54,685 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/A of e07ddca27cd19d46379b6939f22f78c7 into c63f23ca16d24b53a51e051021398738(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:54,685 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:54,685 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/A, priority=13, startTime=1682567694264; duration=0sec 2023-04-27 03:54:54,685 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:54,780 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/3ad38f20722e4ac1b7d826e5173730fa 2023-04-27 03:54:54,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:54,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37824 deadline: 1682567754875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:54,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567754875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:54,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:54,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567754876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:55,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37824 deadline: 1682567755180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:55,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567755180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:55,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567755180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,191 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/6f84409d5e9441cb855d28a0f7a4e123 2023-04-27 03:54:55,601 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/f23a2ba3791048a68be91ce24efe1b71 2023-04-27 03:54:55,605 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/3ad38f20722e4ac1b7d826e5173730fa as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/3ad38f20722e4ac1b7d826e5173730fa 2023-04-27 03:54:55,609 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/3ad38f20722e4ac1b7d826e5173730fa, entries=150, sequenceid=373, filesize=11.9 K 2023-04-27 03:54:55,610 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/6f84409d5e9441cb855d28a0f7a4e123 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/6f84409d5e9441cb855d28a0f7a4e123 2023-04-27 03:54:55,613 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/6f84409d5e9441cb855d28a0f7a4e123, entries=150, sequenceid=373, filesize=11.9 K 2023-04-27 03:54:55,614 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/f23a2ba3791048a68be91ce24efe1b71 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/f23a2ba3791048a68be91ce24efe1b71 2023-04-27 03:54:55,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:55,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567755615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,618 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/f23a2ba3791048a68be91ce24efe1b71, entries=150, sequenceid=373, filesize=11.9 K 2023-04-27 03:54:55,621 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for e07ddca27cd19d46379b6939f22f78c7 in 1249ms, sequenceid=373, compaction requested=false 2023-04-27 03:54:55,621 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:55,621 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:55,621 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:55,621 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:55,621 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,621 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:55,621 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:55,623 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,623 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:55,624 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,624 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:55,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:55,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:55,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f274832[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:55,625 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:55,625 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,626 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:55,626 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,627 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:55,627 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:55,627 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:55,629 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:55,629 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,629 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:55,629 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:55,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:55,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:55,631 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:55,631 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:55,631 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:55,631 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:55,631 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:55,633 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:55,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:55,633 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,633 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:55,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:55,633 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:55,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:55,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:55,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:55,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:55,634 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:55,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:55,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,636 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:55,636 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:55,637 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:55,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:55,637 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:55,637 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:55,637 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,637 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,637 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,637 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:55,637 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:55,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:55,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:55,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:55,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:55,637 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:55,637 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:55,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:55,685 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e07ddca27cd19d46379b6939f22f78c7 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-27 03:54:55,686 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=A 2023-04-27 03:54:55,686 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:55,686 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=B 2023-04-27 03:54:55,686 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:55,686 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=C 2023-04-27 03:54:55,686 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:55,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:55,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:55,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567755710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37824 deadline: 1682567755710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:55,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567755713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,738 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:55,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:55,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:55,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:55,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:55,741 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:55,741 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:55,741 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:55,741 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:55,741 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,741 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:55,743 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,743 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:55,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:55,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:55,743 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,743 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:55,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:55,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:55,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:55,744 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:55,744 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:55,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:55,744 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:55,744 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:55,744 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:55,744 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:55,744 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:55,744 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:55,744 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:55,744 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:55,744 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:55,744 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:55,744 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:55,744 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:55,747 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,747 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:55,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:55,748 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,748 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:55,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:55,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dd150b1[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:55,749 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:55,749 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,751 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,751 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:55,751 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:55,751 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,751 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:55,751 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:55,753 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:55,753 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:55,753 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:55,753 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:55,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:55,755 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:55,755 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:55,755 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:55,755 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:55,755 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:55,757 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:55,757 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,757 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:55,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:55,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:55,758 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:55,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:55,759 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:55,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,762 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,762 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:55,762 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:55,762 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:55,762 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:55,762 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:55,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:55,762 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:55,763 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:55,763 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:55,763 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:55,763 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,763 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:55,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:55,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:55,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:55,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567755814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:55,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37824 deadline: 1682567755814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:55,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567755816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,863 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:55,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:55,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:55,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:55,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:55,866 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:55,866 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:55,866 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:55,866 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:55,867 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,867 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:55,868 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,868 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:55,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:55,868 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,869 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:55,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:55,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:55,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:55,869 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:55,869 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:55,869 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:55,870 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:55,870 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:55,870 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:55,872 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:55,872 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:55,872 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:55,872 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:55,872 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:55,872 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:55,872 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:55,872 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:55,872 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,872 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:55,872 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:55,874 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,874 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:55,874 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:55,875 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:55,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:55,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e02bff8[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:55,876 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:55,876 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,877 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,877 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:55,877 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:55,877 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:55,877 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,877 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:55,879 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,879 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:55,880 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:55,880 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:55,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:55,882 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:55,882 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:55,882 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:55,882 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:55,882 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:55,884 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,884 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,884 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:55,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,885 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:55,886 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,888 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,888 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:55,888 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:55,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:55,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:55,888 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:55,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:55,888 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:55,888 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:55,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:55,889 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,888 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:55,889 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:55,888 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:55,889 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:55,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:55,889 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,989 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:55,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:55,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:55,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:55,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:55,992 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:55,992 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:55,992 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:55,992 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:55,993 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,993 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:55,994 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,994 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:55,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:55,995 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:55,995 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:55,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:55,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:55,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:55,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:55,996 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:55,996 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:55,996 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:55,996 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:55,999 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,999 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:55,999 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,000 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:56,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:56,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1453642[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:56,001 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:56,001 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,002 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,002 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:56,002 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:56,002 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:56,002 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,002 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:56,004 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,004 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:56,004 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:56,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:56,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:56,004 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:56,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:56,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:56,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:56,006 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:56,006 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:56,006 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:56,006 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:56,006 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:56,008 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,008 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,008 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:56,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:56,009 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,010 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,013 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,013 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,013 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,013 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,013 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,013 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:56,013 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:56,013 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:56,013 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:56,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:56,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:56,013 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:56,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:56,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:56,014 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:56,014 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:56,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:56,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37824 deadline: 1682567756017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:56,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567756019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:56,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567756019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,094 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/d2b17b9a768445cfb7bfd397d3867ebf 2023-04-27 03:54:56,114 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:56,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:56,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:56,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:56,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:56,117 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:56,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:56,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:56,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:56,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,119 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,119 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:56,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:56,119 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,119 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:56,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:56,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:56,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:56,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:56,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:56,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:56,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:56,120 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:56,120 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:56,120 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:56,120 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:56,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:56,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:56,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:56,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:56,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,122 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:56,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:56,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:56,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64c68880[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:56,124 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:56,124 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,129 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,129 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,129 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:56,129 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:56,129 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:56,129 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:56,131 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,131 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:56,131 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:56,131 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:56,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:56,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:56,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:56,133 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:56,134 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:56,134 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:56,134 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:56,134 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:56,135 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,135 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:56,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,135 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:56,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:56,136 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:56,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:56,138 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,140 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,140 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:56,141 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:56,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:56,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:56,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:56,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:56,141 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:56,140 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,141 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:56,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:56,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:56,141 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:56,141 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,141 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,141 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,141 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,141 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,241 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:56,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:56,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:56,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:56,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:56,249 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:56,249 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:56,249 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:56,249 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:56,250 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,250 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,251 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:56,251 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:56,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:56,251 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,251 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:56,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:56,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:56,253 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:56,253 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:56,253 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:56,253 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:56,253 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:56,253 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:56,253 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:56,253 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:56,253 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:56,253 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:56,253 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:56,253 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,253 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:56,253 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:56,255 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,255 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:56,255 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,255 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:56,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:56,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53251871[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:56,257 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:56,257 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,258 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,258 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:56,258 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:56,258 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:56,258 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,258 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:56,263 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,263 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:56,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:56,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:56,263 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:56,263 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:56,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:56,266 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:56,266 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:56,266 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:56,266 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:56,266 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:56,267 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,267 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:56,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:56,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,267 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:56,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:56,268 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:56,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:56,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,269 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:56,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,272 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:56,272 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:56,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:56,272 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:56,272 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,272 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,273 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,273 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,273 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,273 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:56,273 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:56,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:56,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:56,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:56,273 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:56,273 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:56,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:56,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567756322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:56,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567756322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:56,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37824 deadline: 1682567756322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,373 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:56,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:56,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:56,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:56,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:56,377 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:56,377 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:56,377 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:56,377 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:56,377 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,377 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,379 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,379 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:56,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:56,379 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,379 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:56,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:56,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:56,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:56,380 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:56,380 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:56,380 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:56,380 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:56,380 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:56,381 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:56,381 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:56,381 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:56,381 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:56,381 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:56,381 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:56,381 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,381 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:56,381 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:56,383 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,383 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:56,383 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:56,383 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:56,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:56,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:56,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:56,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68e39092[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:56,385 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:56,385 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,386 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,386 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:56,386 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:56,386 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:56,386 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,386 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:56,388 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,388 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:56,388 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:56,388 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:56,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:56,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:56,393 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:56,393 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:56,393 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:56,393 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:56,393 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:56,394 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,394 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:56,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,395 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:56,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:56,395 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,401 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,409 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:56,409 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:56,409 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:56,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:56,409 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:56,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:56,409 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:56,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:56,409 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,410 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,410 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,410 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:56,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:56,410 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,410 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:56,410 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:56,505 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/b4403d88c2c44d328a645772bab9fafa 2023-04-27 03:54:56,510 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:56,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:56,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:56,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:56,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:56,513 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:56,513 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:56,513 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:56,513 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:56,514 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,514 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,515 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,515 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:56,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:56,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:56,515 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,515 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:56,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:56,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:56,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:56,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:56,517 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:56,517 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:56,517 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:56,517 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:56,517 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:56,517 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:56,518 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:56,518 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:56,518 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:56,518 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:56,518 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:56,518 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,518 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:56,518 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:56,519 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,519 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:56,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:56,520 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,520 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:56,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:56,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:56,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:56,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:56,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@403c4cea[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:56,521 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:56,521 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,522 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,522 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:56,522 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:56,522 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:56,522 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,522 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:56,527 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,527 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:56,527 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:56,527 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:56,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:56,529 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:56,529 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:56,529 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:56,530 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:56,530 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:56,531 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,531 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:56,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,531 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:56,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:56,532 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:56,533 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,535 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:56,535 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:56,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:56,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:56,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:56,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:56,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:56,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:56,535 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:56,535 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,535 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:56,535 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,535 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:56,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:56,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,636 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:56,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:56,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:56,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:56,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:56,644 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:56,644 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:56,644 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:56,644 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:56,644 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,644 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,645 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,645 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:56,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:56,646 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,646 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:56,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:56,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:56,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:56,647 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:56,647 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:56,647 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:56,647 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:56,647 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:56,647 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:56,647 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:56,648 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:56,648 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:56,648 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:56,648 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:56,648 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,648 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:56,648 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:56,651 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,651 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:56,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:56,651 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,651 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:56,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:56,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:56,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:56,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@106e4ee5[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:56,653 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:56,653 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,654 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,654 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,654 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:56,654 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:56,654 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:56,654 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:56,656 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:56,656 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:56,656 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,656 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:56,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:56,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:56,659 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:56,659 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:56,659 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:56,659 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:56,659 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:56,661 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,661 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:56,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,661 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:56,661 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:56,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,663 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:56,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,667 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,667 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:56,667 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:56,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:56,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:56,667 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:56,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:56,667 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:56,667 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:56,667 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,667 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:56,667 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:56,667 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,667 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,667 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,767 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:56,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:56,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:56,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:56,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:56,770 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:56,771 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:56,771 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:56,771 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:56,771 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,771 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,772 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:56,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:56,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:56,772 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,773 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,773 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:56,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:56,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:56,774 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:56,774 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:56,774 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:56,774 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:56,774 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:56,774 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:56,774 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:56,774 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:56,774 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:56,774 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:56,774 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:56,774 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:56,774 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:56,774 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,774 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:56,774 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:56,776 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,776 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:56,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:56,776 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,776 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:56,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:56,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:56,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:56,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:56,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bf32a09[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:56,778 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:56,778 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,779 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,779 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,779 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:56,779 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:56,779 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:56,779 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:56,782 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:56,782 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:56,782 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,782 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:56,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:56,784 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:56,784 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:56,784 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:56,784 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:56,784 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:56,786 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,786 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:56,786 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:56,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,787 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,788 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:56,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,791 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:56,791 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:56,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:56,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:56,791 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:56,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:56,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:56,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:56,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:56,791 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:56,791 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:56,796 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,796 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,796 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,796 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,796 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,796 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,796 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:56,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37824 deadline: 1682567756824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:56,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:56,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567756826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567756826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,891 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:56,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:56,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:56,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:56,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:56,895 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:56,895 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:56,895 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:56,895 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:56,895 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,895 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,896 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:56,896 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,897 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,897 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:56,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:56,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:56,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:56,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:56,899 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:56,899 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:56,899 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:56,899 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:56,899 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:56,899 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:56,899 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:56,899 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:56,899 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:56,899 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:56,899 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:56,899 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,899 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:56,899 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:56,902 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,902 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:56,902 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,903 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:56,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:56,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fe050e5[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:56,904 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:56,904 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,905 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,905 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:56,905 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:56,905 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:56,905 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,905 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:56,907 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,907 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:56,907 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:56,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:56,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:56,907 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:56,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:56,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:56,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:56,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:56,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:56,909 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:56,909 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:56,909 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:56,909 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:56,909 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:56,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:56,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:56,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:56,911 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,912 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,917 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,917 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:56,917 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:56,917 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:56,917 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:56,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:56,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:56,917 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:56,917 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:56,917 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,917 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:56,917 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:56,917 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:56,917 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:56,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:56,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:56,919 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/e196efe8bca34d9391364c9a8f79b8fa 2023-04-27 03:54:56,923 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/d2b17b9a768445cfb7bfd397d3867ebf as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/d2b17b9a768445cfb7bfd397d3867ebf 2023-04-27 03:54:56,926 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/d2b17b9a768445cfb7bfd397d3867ebf, entries=150, sequenceid=386, filesize=11.9 K 2023-04-27 03:54:56,927 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/b4403d88c2c44d328a645772bab9fafa as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/b4403d88c2c44d328a645772bab9fafa 2023-04-27 03:54:56,932 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/b4403d88c2c44d328a645772bab9fafa, entries=100, sequenceid=386, filesize=9.5 K 2023-04-27 03:54:56,933 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/e196efe8bca34d9391364c9a8f79b8fa as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/e196efe8bca34d9391364c9a8f79b8fa 2023-04-27 03:54:56,939 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/e196efe8bca34d9391364c9a8f79b8fa, entries=100, sequenceid=386, filesize=9.5 K 2023-04-27 03:54:56,940 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for e07ddca27cd19d46379b6939f22f78c7 in 1255ms, sequenceid=386, compaction requested=true 2023-04-27 03:54:56,940 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:56,940 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:56,940 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:56,940 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:56,940 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:56,940 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:56,942 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37372 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:56,942 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/A is initiating minor compaction (all files) 2023-04-27 03:54:56,942 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34932 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:56,942 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/A in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:56,942 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/B is initiating minor compaction (all files) 2023-04-27 03:54:56,942 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/c63f23ca16d24b53a51e051021398738, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/3ad38f20722e4ac1b7d826e5173730fa, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/d2b17b9a768445cfb7bfd397d3867ebf] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=36.5 K 2023-04-27 03:54:56,942 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/B in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:56,942 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/61934ef7a6d8418abcf8947210fc8bee, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/6f84409d5e9441cb855d28a0f7a4e123, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/b4403d88c2c44d328a645772bab9fafa] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=34.1 K 2023-04-27 03:54:56,943 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting c63f23ca16d24b53a51e051021398738, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1682567693402 2023-04-27 03:54:56,943 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 61934ef7a6d8418abcf8947210fc8bee, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1682567693402 2023-04-27 03:54:56,943 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 3ad38f20722e4ac1b7d826e5173730fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1682567693484 2023-04-27 03:54:56,943 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 6f84409d5e9441cb855d28a0f7a4e123, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1682567693484 2023-04-27 03:54:56,943 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting d2b17b9a768445cfb7bfd397d3867ebf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1682567694566 2023-04-27 03:54:56,944 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting b4403d88c2c44d328a645772bab9fafa, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1682567694569 2023-04-27 03:54:56,951 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#A#compaction#285 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:56,951 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#B#compaction#286 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:56,961 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/eadc85396c71421f8abdb2076f3b7ecc as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/eadc85396c71421f8abdb2076f3b7ecc 2023-04-27 03:54:56,962 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/9e94dc62e3f448ad9b4b33b49a172182 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/9e94dc62e3f448ad9b4b33b49a172182 2023-04-27 03:54:56,966 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/B of e07ddca27cd19d46379b6939f22f78c7 into 9e94dc62e3f448ad9b4b33b49a172182(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:56,967 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:56,967 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/B, priority=13, startTime=1682567696940; duration=0sec 2023-04-27 03:54:56,966 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/A of e07ddca27cd19d46379b6939f22f78c7 into eadc85396c71421f8abdb2076f3b7ecc(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:56,967 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:56,967 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:56,967 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:56,967 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/A, priority=13, startTime=1682567696940; duration=0sec 2023-04-27 03:54:56,967 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:56,968 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34898 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:56,968 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/C is initiating minor compaction (all files) 2023-04-27 03:54:56,968 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/C in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:56,968 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/04a66e0639e14d329a583d5187d6b41d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/f23a2ba3791048a68be91ce24efe1b71, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/e196efe8bca34d9391364c9a8f79b8fa] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=34.1 K 2023-04-27 03:54:56,968 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 04a66e0639e14d329a583d5187d6b41d, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1682567693402 2023-04-27 03:54:56,969 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting f23a2ba3791048a68be91ce24efe1b71, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1682567693484 2023-04-27 03:54:56,969 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting e196efe8bca34d9391364c9a8f79b8fa, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1682567694569 2023-04-27 03:54:56,975 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#C#compaction#287 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:56,984 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/ed42f2362ed44bfcb9586c15ec9a2da9 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/ed42f2362ed44bfcb9586c15ec9a2da9 2023-04-27 03:54:56,989 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/C of e07ddca27cd19d46379b6939f22f78c7 into ed42f2362ed44bfcb9586c15ec9a2da9(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:56,989 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:56,989 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/C, priority=13, startTime=1682567696940; duration=0sec 2023-04-27 03:54:56,989 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:57,017 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:57,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:57,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:57,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:57,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:57,021 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:57,021 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:57,021 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:57,021 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:57,021 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,021 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,023 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,023 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:57,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:57,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:57,023 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,023 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:57,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:57,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:57,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:57,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:57,024 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:57,024 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:57,024 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:57,024 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:57,024 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:57,024 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:57,024 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:57,025 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e07ddca27cd19d46379b6939f22f78c7 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-27 03:54:57,025 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=A 2023-04-27 03:54:57,025 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:57,025 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=B 2023-04-27 03:54:57,025 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:57,025 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=C 2023-04-27 03:54:57,025 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:57,434 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/21ec161f247a4c5894fc791752d2ab21 2023-04-27 03:54:57,444 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/a520c0752d174bff8dc2411b4605ff49 2023-04-27 03:54:57,454 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/5b85a5cdc37e466e885113a2960c3064 2023-04-27 03:54:57,457 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/21ec161f247a4c5894fc791752d2ab21 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/21ec161f247a4c5894fc791752d2ab21 2023-04-27 03:54:57,461 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/21ec161f247a4c5894fc791752d2ab21, entries=150, sequenceid=409, filesize=11.9 K 2023-04-27 03:54:57,462 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/a520c0752d174bff8dc2411b4605ff49 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/a520c0752d174bff8dc2411b4605ff49 2023-04-27 03:54:57,465 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/a520c0752d174bff8dc2411b4605ff49, entries=150, sequenceid=409, filesize=11.9 K 2023-04-27 03:54:57,466 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/5b85a5cdc37e466e885113a2960c3064 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/5b85a5cdc37e466e885113a2960c3064 2023-04-27 03:54:57,469 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/5b85a5cdc37e466e885113a2960c3064, entries=150, sequenceid=409, filesize=11.9 K 2023-04-27 03:54:57,470 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for e07ddca27cd19d46379b6939f22f78c7 in 445ms, sequenceid=409, compaction requested=false 2023-04-27 03:54:57,470 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:57,470 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:57,470 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:57,470 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:57,470 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,470 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:57,470 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:57,472 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:57,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:57,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:57,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d1904ef[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:57,474 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:57,474 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,475 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,476 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,476 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:57,476 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:57,476 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:57,476 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:57,479 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,479 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:57,479 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:57,479 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:57,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:57,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:57,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:57,481 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:57,481 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:57,481 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:57,481 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:57,481 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:57,483 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,483 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:57,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,483 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:57,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:57,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,485 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:57,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,488 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:57,488 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,488 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,488 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,488 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:57,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:57,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:57,488 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:57,488 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,488 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,488 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,488 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:57,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:57,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:57,489 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:57,489 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:57,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:57,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:57,589 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:57,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:57,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:57,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:57,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:57,592 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:57,592 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:57,592 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:57,592 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:57,593 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,593 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,594 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:57,594 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:57,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:57,595 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,595 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:57,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:57,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:57,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:57,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:57,596 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:57,596 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:57,596 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:57,596 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:57,596 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:57,596 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:57,596 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:57,596 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:57,596 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:57,596 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:57,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:57,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:57,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:57,599 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,599 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:57,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:57,599 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,599 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:57,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:57,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:57,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:57,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:57,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fe27ccf[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:57,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:57,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,602 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:57,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:57,602 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:57,602 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:57,604 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:57,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:57,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:57,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:57,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:57,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:57,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:57,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:57,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:57,607 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:57,607 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:57,607 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:57,607 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:57,607 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:57,608 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,608 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:57,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:57,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,609 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:57,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:57,609 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:57,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:57,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:57,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,610 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:57,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,613 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:57,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:57,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:57,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:57,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:57,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:57,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:57,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:57,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,614 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,614 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:57,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:57,614 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,614 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:57,614 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:57,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:57,634 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e07ddca27cd19d46379b6939f22f78c7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:54:57,634 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=A 2023-04-27 03:54:57,635 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:57,635 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=B 2023-04-27 03:54:57,635 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:57,635 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=C 2023-04-27 03:54:57,635 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:57,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567697655, max timestamp=1682567697665Num uniques 859; 2023-04-27 03:54:57,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:54:57,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567697655, max timestamp=1682567697665Num uniques 859; 2023-04-27 03:54:57,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567697655, max timestamp=1682567697665Num uniques 859; 2023-04-27 03:54:57,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:54:57,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:54:57,714 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:57,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:57,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:57,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567757712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:57,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37876 deadline: 1682567757713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:57,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:57,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:57,718 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:57,718 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:57,718 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:57,718 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:57,718 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,718 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,719 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,719 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:57,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:57,720 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,720 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:57,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:57,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:57,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:57,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:57,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:57,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:57,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:57,721 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:57,721 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:57,721 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:57,722 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:57,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:57,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:57,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:57,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:57,723 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:57,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:57,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:57,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:57,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:57,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:57,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d33a658[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:57,725 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:57,725 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,726 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,726 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:57,726 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:57,726 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:57,726 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,726 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:57,729 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:57,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:57,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:57,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:57,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:57,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:57,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:57,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:57,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:57,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:57,731 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:57,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:57,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:57,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:57,731 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:57,733 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,733 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:57,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,733 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:57,733 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:57,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,734 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:57,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,739 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:57,739 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:57,739 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:57,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:57,739 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,739 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:57,739 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:57,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:57,739 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,739 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:57,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:57,739 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,740 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,740 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:57,740 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:57,817 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:57,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567757817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:57,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37876 deadline: 1682567757817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:57,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37824 deadline: 1682567757826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:57,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37840 deadline: 1682567757836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:57,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37856 deadline: 1682567757836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,840 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:57,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:57,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:57,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:57,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:57,843 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:57,843 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:57,843 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:57,843 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:57,843 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,844 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,846 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,846 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:57,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:57,846 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,846 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:57,847 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:57,847 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:57,847 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:57,847 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:57,847 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:57,847 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:57,847 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:57,848 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:57,848 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:57,848 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:57,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:57,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:57,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:57,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:57,849 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:57,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:57,850 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,850 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:57,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:57,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:57,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:57,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:57,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30c9ac63[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:57,851 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:57,851 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,853 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,853 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:57,853 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,853 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:57,853 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:57,853 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:57,854 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,854 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:57,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:57,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:57,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:57,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:57,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:57,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:57,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:57,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:57,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:57,857 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:57,857 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:57,857 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:57,857 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:57,857 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:57,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:57,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:57,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:57,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:57,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:57,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:57,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,860 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,861 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:57,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,863 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,863 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,863 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,863 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:57,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:57,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:57,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:57,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:57,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:57,864 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:57,864 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:57,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:57,964 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:57,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:57,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:57,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:57,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:57,967 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:57,968 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:57,968 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:57,968 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:57,968 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,968 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,969 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:57,969 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:57,969 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,970 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:57,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:57,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:57,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:57,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:57,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:57,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:57,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:57,971 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:57,971 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:57,971 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:57,971 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:57,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:57,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:57,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:57,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:57,973 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,973 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:57,973 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,973 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:57,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:57,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4502e8e9[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:57,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:57,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,976 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:57,977 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:57,977 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:57,977 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:57,978 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:57,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:57,978 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:57,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:57,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:57,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:57,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:57,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:57,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:57,981 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:57,981 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:57,981 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:57,981 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:57,981 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:57,983 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,983 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,983 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:57,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,983 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:57,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:57,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:57,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:57,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:57,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:57,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,986 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:57,986 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:57,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:57,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:57,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:57,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:57,987 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:57,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:57,987 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:57,987 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:57,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:57,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:57,987 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,987 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,987 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:57,987 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:57,987 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,987 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:57,987 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:58,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567758019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:58,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37876 deadline: 1682567758021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,054 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/e4ca75916940456daa370575086c1066 2023-04-27 03:54:58,072 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/f0bce6ca8b464defaa673bd7d0cf4710 2023-04-27 03:54:58,087 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:58,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:58,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:58,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:58,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:58,090 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:58,090 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:58,090 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:58,090 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:58,091 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,091 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:58,093 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:58,093 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:58,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:58,094 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,094 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:58,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:58,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:58,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:58,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:58,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:58,095 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:58,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:58,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:58,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:58,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:58,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:58,095 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:58,095 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:58,095 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:58,095 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:58,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:58,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:58,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,096 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:58,096 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:58,097 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,097 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:58,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:58,097 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,097 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:58,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:58,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:58,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:58,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:58,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:58,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9caf082[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:58,099 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:58,099 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,100 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,100 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:58,100 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:58,100 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:58,100 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,100 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:58,101 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:58,101 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:58,101 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,101 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:58,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:58,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:58,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:58,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:58,104 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:58,104 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:58,104 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:58,104 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:58,104 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:58,106 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,106 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:58,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:58,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,106 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:58,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:58,107 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:58,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:58,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:58,108 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:58,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:58,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:58,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:58,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:58,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:58,110 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:58,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:58,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:58,111 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:58,111 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:58,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:58,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:58,111 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:58,111 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:58,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:58,211 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:58,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:58,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:58,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:58,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:58,215 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:58,215 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:58,215 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:58,215 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:58,215 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,215 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:58,218 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:58,218 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:58,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:58,219 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,219 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:58,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:58,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:58,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:58,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:58,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:58,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:58,220 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:58,220 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:58,220 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:58,220 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:58,220 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:58,220 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:58,220 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:58,220 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:58,220 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:58,220 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:58,220 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:58,220 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,220 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:58,221 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:58,223 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,223 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:58,223 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,223 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:58,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:58,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:58,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:58,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:58,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39fb9685[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:58,224 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:58,225 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,226 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,226 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:58,226 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:58,226 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:58,226 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,226 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:58,227 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,227 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:58,227 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:58,227 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:58,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:58,229 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:58,229 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:58,229 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:58,229 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:58,229 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:58,231 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,231 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,231 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:58,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:58,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:58,231 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:58,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:58,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:58,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:58,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:58,232 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:58,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:58,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:58,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,236 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,236 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:58,236 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:58,236 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:58,236 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:58,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:58,236 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,236 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,236 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,236 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:58,236 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:58,236 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:58,236 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:58,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:58,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:58,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:58,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:58,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:58,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567758322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:58,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37876 deadline: 1682567758326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,336 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:58,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:58,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:58,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:58,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:58,339 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:58,340 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:58,340 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:58,340 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:58,340 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,340 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:58,341 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:58,341 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:58,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:58,342 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,342 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:58,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:58,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:58,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:58,342 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:58,343 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:58,343 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:58,343 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:58,343 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:58,343 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:58,343 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:58,343 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:58,343 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:58,343 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:58,343 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:58,343 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:58,343 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:58,343 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:58,343 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,343 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:58,343 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:58,345 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,345 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:58,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:58,345 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,345 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:58,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:58,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:58,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:58,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@442c8f6d[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:58,346 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:58,347 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,348 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,348 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:58,348 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:58,348 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,348 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:58,348 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:58,350 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:58,351 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:58,351 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:58,351 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:58,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:58,353 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:58,353 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:58,353 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:58,353 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:58,353 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:58,355 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,355 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,355 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:58,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:58,355 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:58,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:58,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:58,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:58,358 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,358 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:58,359 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:58,359 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:58,359 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:58,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:58,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:58,359 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,359 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,359 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,359 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,359 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:58,359 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:58,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:58,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:58,359 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:58,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:58,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:58,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:58,459 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:58,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:58,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:58,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:58,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:58,463 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:58,463 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:58,463 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:58,463 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:58,463 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,463 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:58,464 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:58,465 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:58,465 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,465 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:58,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:58,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:58,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:58,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:58,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:58,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:58,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:58,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:58,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:58,466 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:58,466 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:58,466 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:58,467 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:58,467 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:58,467 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:58,467 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,467 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:58,467 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:58,469 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,469 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:58,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:58,470 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,470 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:58,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:58,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16bb62a1[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:58,471 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:58,471 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,472 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:58,472 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,472 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:58,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:58,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:58,474 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:58,474 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:58,474 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:58,474 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:58,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:58,477 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:58,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:58,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:58,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:58,477 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:58,478 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,478 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:58,478 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:58,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:58,479 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:58,480 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:58,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:58,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,482 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/27170bd262034d88ae3497fc542a851f 2023-04-27 03:54:58,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:58,485 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:58,485 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:58,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:58,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:58,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:58,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:58,485 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:58,485 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:58,485 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:58,485 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,485 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,485 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:58,485 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,485 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:58,485 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:58,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:58,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:58,486 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/e4ca75916940456daa370575086c1066 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/e4ca75916940456daa370575086c1066 2023-04-27 03:54:58,489 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/e4ca75916940456daa370575086c1066, entries=100, sequenceid=420, filesize=9.5 K 2023-04-27 03:54:58,490 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/f0bce6ca8b464defaa673bd7d0cf4710 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/f0bce6ca8b464defaa673bd7d0cf4710 2023-04-27 03:54:58,493 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/f0bce6ca8b464defaa673bd7d0cf4710, entries=100, sequenceid=420, filesize=9.5 K 2023-04-27 03:54:58,494 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/27170bd262034d88ae3497fc542a851f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/27170bd262034d88ae3497fc542a851f 2023-04-27 03:54:58,497 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/27170bd262034d88ae3497fc542a851f, entries=100, sequenceid=420, filesize=9.5 K 2023-04-27 03:54:58,498 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e07ddca27cd19d46379b6939f22f78c7 in 864ms, sequenceid=420, compaction requested=true 2023-04-27 03:54:58,498 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:58,498 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:58,498 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:58,498 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:58,498 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:58,498 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:58,499 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35034 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:58,499 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/A is initiating minor compaction (all files) 2023-04-27 03:54:58,499 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35034 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:58,499 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/A in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:58,499 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/B is initiating minor compaction (all files) 2023-04-27 03:54:58,499 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/eadc85396c71421f8abdb2076f3b7ecc, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/21ec161f247a4c5894fc791752d2ab21, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/e4ca75916940456daa370575086c1066] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=34.2 K 2023-04-27 03:54:58,499 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/B in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:58,499 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/9e94dc62e3f448ad9b4b33b49a172182, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/a520c0752d174bff8dc2411b4605ff49, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/f0bce6ca8b464defaa673bd7d0cf4710] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=34.2 K 2023-04-27 03:54:58,500 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting eadc85396c71421f8abdb2076f3b7ecc, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1682567693494 2023-04-27 03:54:58,500 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 9e94dc62e3f448ad9b4b33b49a172182, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1682567693494 2023-04-27 03:54:58,500 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 21ec161f247a4c5894fc791752d2ab21, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1682567695704 2023-04-27 03:54:58,500 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting a520c0752d174bff8dc2411b4605ff49, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1682567695704 2023-04-27 03:54:58,500 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting e4ca75916940456daa370575086c1066, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1682567697627 2023-04-27 03:54:58,500 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting f0bce6ca8b464defaa673bd7d0cf4710, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1682567697627 2023-04-27 03:54:58,506 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#B#compaction#294 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:58,508 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#A#compaction#295 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:58,521 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/1edf0e6218114d89a5fb5a168fb4bcd4 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/1edf0e6218114d89a5fb5a168fb4bcd4 2023-04-27 03:54:58,524 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/e176ebe95f6540e3957222d93e620a5a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/e176ebe95f6540e3957222d93e620a5a 2023-04-27 03:54:58,526 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/B of e07ddca27cd19d46379b6939f22f78c7 into 1edf0e6218114d89a5fb5a168fb4bcd4(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:58,526 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:58,526 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/B, priority=13, startTime=1682567698498; duration=0sec 2023-04-27 03:54:58,526 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:54:58,526 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:54:58,527 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35000 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:54:58,528 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e07ddca27cd19d46379b6939f22f78c7/C is initiating minor compaction (all files) 2023-04-27 03:54:58,528 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e07ddca27cd19d46379b6939f22f78c7/C in TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:58,528 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/ed42f2362ed44bfcb9586c15ec9a2da9, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/5b85a5cdc37e466e885113a2960c3064, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/27170bd262034d88ae3497fc542a851f] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp, totalSize=34.2 K 2023-04-27 03:54:58,528 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting ed42f2362ed44bfcb9586c15ec9a2da9, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1682567693494 2023-04-27 03:54:58,528 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/A of e07ddca27cd19d46379b6939f22f78c7 into e176ebe95f6540e3957222d93e620a5a(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:58,528 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:58,528 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/A, priority=13, startTime=1682567698498; duration=0sec 2023-04-27 03:54:58,528 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 5b85a5cdc37e466e885113a2960c3064, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1682567695704 2023-04-27 03:54:58,528 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:58,529 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 27170bd262034d88ae3497fc542a851f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1682567697627 2023-04-27 03:54:58,534 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e07ddca27cd19d46379b6939f22f78c7#C#compaction#296 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:54:58,585 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:58,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:58,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:54:58,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:54:58,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:54:58,588 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:54:58,589 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:58,589 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:54:58,589 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:54:58,589 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,589 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:58,590 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:58,590 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:58,590 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,590 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:54:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:58,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:54:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:58,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:54:58,591 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:54:58,591 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:54:58,591 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:54:58,591 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:54:58,591 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:54:58,592 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:54:58,592 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:54:58,592 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:58,592 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. started... 2023-04-27 03:54:58,592 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e07ddca27cd19d46379b6939f22f78c7 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-27 03:54:58,592 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=A 2023-04-27 03:54:58,592 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:58,592 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=B 2023-04-27 03:54:58,592 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:58,592 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=C 2023-04-27 03:54:58,592 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:54:58,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:58,827 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. as already flushing 2023-04-27 03:54:58,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:58,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37876 deadline: 1682567758838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:58,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567758838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,944 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/98edde672eba4a1e8d4baebdee4acbb6 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/98edde672eba4a1e8d4baebdee4acbb6 2023-04-27 03:54:58,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:58,946 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:58,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37876 deadline: 1682567758943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567758943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:58,949 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e07ddca27cd19d46379b6939f22f78c7/C of e07ddca27cd19d46379b6939f22f78c7 into 98edde672eba4a1e8d4baebdee4acbb6(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:54:58,949 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:58,949 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7., storeName=e07ddca27cd19d46379b6939f22f78c7/C, priority=13, startTime=1682567698498; duration=0sec 2023-04-27 03:54:58,949 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:54:59,000 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/76bb0d304def43959e3964d249b260a3 2023-04-27 03:54:59,010 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/81c2c7b82462431abd8d060e885953c3 2023-04-27 03:54:59,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:59,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37796 deadline: 1682567759147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:59,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:54:59,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37876 deadline: 1682567759148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:59,368 DEBUG [Thread-1719] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12115052 to 127.0.0.1:55020 2023-04-27 03:54:59,368 DEBUG [Thread-1719] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:59,369 DEBUG [Thread-1717] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x087b3095 to 127.0.0.1:55020 2023-04-27 03:54:59,369 DEBUG [Thread-1717] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:59,370 DEBUG [Thread-1725] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f8ff99f to 127.0.0.1:55020 2023-04-27 03:54:59,370 DEBUG [Thread-1723] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6fcbaba8 to 127.0.0.1:55020 2023-04-27 03:54:59,370 DEBUG [Thread-1725] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:59,370 DEBUG [Thread-1723] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:59,371 DEBUG [Thread-1721] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e65578e to 127.0.0.1:55020 2023-04-27 03:54:59,371 DEBUG [Thread-1721] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:59,420 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/c5b176f815cc4c8c843f61e45b40f036 2023-04-27 03:54:59,423 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/76bb0d304def43959e3964d249b260a3 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/76bb0d304def43959e3964d249b260a3 2023-04-27 03:54:59,426 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/76bb0d304def43959e3964d249b260a3, entries=150, sequenceid=458, filesize=11.9 K 2023-04-27 03:54:59,427 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/81c2c7b82462431abd8d060e885953c3 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/81c2c7b82462431abd8d060e885953c3 2023-04-27 03:54:59,430 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/81c2c7b82462431abd8d060e885953c3, entries=150, sequenceid=458, filesize=11.9 K 2023-04-27 03:54:59,430 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/c5b176f815cc4c8c843f61e45b40f036 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/c5b176f815cc4c8c843f61e45b40f036 2023-04-27 03:54:59,434 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/c5b176f815cc4c8c843f61e45b40f036, entries=150, sequenceid=458, filesize=11.9 K 2023-04-27 03:54:59,434 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for e07ddca27cd19d46379b6939f22f78c7 in 842ms, sequenceid=458, compaction requested=false 2023-04-27 03:54:59,434 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:54:59,434 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:54:59,434 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:54:59,434 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:54:59,434 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:59,435 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:54:59,435 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:54:59,438 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:59,438 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:59,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:59,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:59,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:59,438 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:59,438 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:54:59,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:59,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:59,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:59,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:59,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:59,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:54:59,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48abde14[Count = 0] remaining members to acquire global barrier 2023-04-27 03:54:59,440 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:54:59,440 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:59,444 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:59,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:59,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:54:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:54:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:54:59,444 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:59,444 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:54:59,445 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:54:59,445 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:59,445 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:59,445 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:54:59,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:59,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:59,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:59,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:59,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:59,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:59,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:59,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:59,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:59,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:59,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:54:59,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:54:59,448 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:54:59,448 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:54:59,448 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:54:59,448 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:54:59,448 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:54:59,449 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:59,449 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:59,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:59,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:54:59,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:54:59,449 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:59,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:59,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:59,450 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:54:59,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:54:59,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:59,451 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:59,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:59,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:54:59,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:54:59,451 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e07ddca27cd19d46379b6939f22f78c7 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-27 03:54:59,451 DEBUG [Thread-1710] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7480f9da to 127.0.0.1:55020 2023-04-27 03:54:59,451 DEBUG [Thread-1710] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:59,451 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=A 2023-04-27 03:54:59,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:54:59,451 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:59,452 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=B 2023-04-27 03:54:59,452 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:59,452 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=C 2023-04-27 03:54:59,452 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:54:59,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:59,453 DEBUG [Thread-1714] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x104dbf4e to 127.0.0.1:55020 2023-04-27 03:54:59,453 DEBUG [Thread-1714] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:59,454 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:59,454 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:59,454 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:54:59,454 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:59,454 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:54:59,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:54:59,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:54:59,454 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:59,454 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:54:59,454 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:54:59,455 DEBUG [Thread-1716] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:54:59,455 DEBUG [Thread-1716] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:54:59,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:54:59,456 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:54:59,456 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:54:59,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:54:59,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:54:59,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:54:59,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:54:59,555 DEBUG [Thread-1716] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:54:59,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:54:59,850 DEBUG [Thread-1706] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00ca898c to 127.0.0.1:55020 2023-04-27 03:54:59,850 DEBUG [Thread-1706] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:59,851 DEBUG [Thread-1708] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6585a70a to 127.0.0.1:55020 2023-04-27 03:54:59,851 DEBUG [Thread-1708] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:59,857 DEBUG [Thread-1712] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a24d617 to 127.0.0.1:55020 2023-04-27 03:54:59,858 DEBUG [Thread-1712] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:59,858 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-27 03:54:59,858 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 64 2023-04-27 03:54:59,858 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 78 2023-04-27 03:54:59,858 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 121 2023-04-27 03:54:59,858 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 83 2023-04-27 03:54:59,858 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 26 2023-04-27 03:54:59,858 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-27 03:54:59,858 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(398): read 5873 2023-04-27 03:54:59,858 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(398): read 5869 2023-04-27 03:54:59,858 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(398): read 5906 2023-04-27 03:54:59,858 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(398): read 5878 2023-04-27 03:54:59,858 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(398): read 5886 2023-04-27 03:54:59,858 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-27 03:54:59,858 INFO [Listener at localhost/33933] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-27 03:54:59,859 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x262f4570 to 127.0.0.1:55020 2023-04-27 03:54:59,859 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/e01d2121e40a425d912b4bcd214497e6 2023-04-27 03:54:59,859 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:54:59,859 INFO [Listener at localhost/33933] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-27 03:54:59,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-27 03:54:59,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-27 03:54:59,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-27 03:54:59,864 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567699864"}]},"ts":"1682567699864"} 2023-04-27 03:54:59,865 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-27 03:54:59,866 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-27 03:54:59,867 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e07ddca27cd19d46379b6939f22f78c7, UNASSIGN}] 2023-04-27 03:54:59,868 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e07ddca27cd19d46379b6939f22f78c7, UNASSIGN 2023-04-27 03:54:59,869 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=e07ddca27cd19d46379b6939f22f78c7, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:54:59,869 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567699869"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682567699869"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682567699869"}]},"ts":"1682567699869"} 2023-04-27 03:54:59,870 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305}] 2023-04-27 03:54:59,872 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/0579550a47ba4b7785e1f8effb55e156 2023-04-27 03:54:59,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-27 03:55:00,023 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:55:00,023 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e07ddca27cd19d46379b6939f22f78c7, disabling compactions & flushes 2023-04-27 03:55:00,023 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:55:00,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-27 03:55:00,286 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/32b2e56a52d04a898513f3e653814fd5 2023-04-27 03:55:00,289 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/e01d2121e40a425d912b4bcd214497e6 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/e01d2121e40a425d912b4bcd214497e6 2023-04-27 03:55:00,292 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/e01d2121e40a425d912b4bcd214497e6, entries=150, sequenceid=470, filesize=11.9 K 2023-04-27 03:55:00,293 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/0579550a47ba4b7785e1f8effb55e156 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/0579550a47ba4b7785e1f8effb55e156 2023-04-27 03:55:00,296 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/0579550a47ba4b7785e1f8effb55e156, entries=150, sequenceid=470, filesize=11.9 K 2023-04-27 03:55:00,297 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/32b2e56a52d04a898513f3e653814fd5 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/32b2e56a52d04a898513f3e653814fd5 2023-04-27 03:55:00,300 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/32b2e56a52d04a898513f3e653814fd5, entries=150, sequenceid=470, filesize=11.9 K 2023-04-27 03:55:00,300 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=26.84 KB/27480 for e07ddca27cd19d46379b6939f22f78c7 in 849ms, sequenceid=470, compaction requested=true 2023-04-27 03:55:00,300 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:55:00,300 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:55:00,301 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:00,301 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:55:00,301 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. because compaction request was cancelled 2023-04-27 03:55:00,301 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. after waiting 0 ms 2023-04-27 03:55:00,301 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:00,301 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:55:00,301 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. because compaction request was cancelled 2023-04-27 03:55:00,301 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing e07ddca27cd19d46379b6939f22f78c7 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-27 03:55:00,301 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. because compaction request was cancelled 2023-04-27 03:55:00,301 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:00,301 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=A 2023-04-27 03:55:00,301 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:00,301 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=B 2023-04-27 03:55:00,301 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:00,301 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e07ddca27cd19d46379b6939f22f78c7, store=C 2023-04-27 03:55:00,301 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:00,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-27 03:55:00,707 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/85d845842ef044298289a1dcadeea301 2023-04-27 03:55:00,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-27 03:55:01,116 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/6778a84c5f29401dba0efcb7c5089981 2023-04-27 03:55:01,524 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/43f3667ae097434883dade6498ae29c9 2023-04-27 03:55:01,528 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/A/85d845842ef044298289a1dcadeea301 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/85d845842ef044298289a1dcadeea301 2023-04-27 03:55:01,530 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/85d845842ef044298289a1dcadeea301, entries=150, sequenceid=477, filesize=11.9 K 2023-04-27 03:55:01,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/B/6778a84c5f29401dba0efcb7c5089981 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/6778a84c5f29401dba0efcb7c5089981 2023-04-27 03:55:01,534 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/6778a84c5f29401dba0efcb7c5089981, entries=150, sequenceid=477, filesize=11.9 K 2023-04-27 03:55:01,534 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/.tmp/C/43f3667ae097434883dade6498ae29c9 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/43f3667ae097434883dade6498ae29c9 2023-04-27 03:55:01,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/43f3667ae097434883dade6498ae29c9, entries=150, sequenceid=477, filesize=11.9 K 2023-04-27 03:55:01,538 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for e07ddca27cd19d46379b6939f22f78c7 in 1237ms, sequenceid=477, compaction requested=true 2023-04-27 03:55:01,560 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/1defe950142546b7acdc141d69d1f2a0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/736cc4ad1c4e46ce88c61b01eba7af34, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/249d0f4e024b48d4acdd7613bad9d86c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/a892a37d297f473bb5feeac38963586f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/1c56857fd3334b388a6e075e21a2a624, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/bb9da04d8b684408b943cdcb7e5c77cd, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/4ee12e19ebb34b4ab45b7e932ff97f26, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/93722a2a9be5466cb3425a7ad982983f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/301cb37987bc4d708d855775b634ea1d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/c2e96bb6722b473e9a3627648c8c2370, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/d70e4141fb61492f934ab6df75dcdfb6, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/9607be49a68246bfb48d22ee71e0de96, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/6ab831af0bba4d2ab74cae0d68630b7a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/ead45b63cc24402e8e914283c4e6f387, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/5ac1f73a478c439f95381e70e18a8a2b, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/a50e26ef347f49f1810f9b2cc657ca71, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/52587078d9c244c28f2da9695e910002, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/f5368c35d39245d98412916eda6146c8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/07f133281b5744e09c2856ff1dd71d7e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/44b8721facbc40c1a59e8f1a2c4f9ba2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/d4c8ca7859e64ae1a92e6d22abcd5f5a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/8845cb603fd148a5b3efc527d8544388, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/d124f23070bc430abbb6fa68c91a761a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/ccd26d3eb8a548baa245614f4027e55f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/c63f23ca16d24b53a51e051021398738, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/3ad38f20722e4ac1b7d826e5173730fa, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/eadc85396c71421f8abdb2076f3b7ecc, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/d2b17b9a768445cfb7bfd397d3867ebf, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/21ec161f247a4c5894fc791752d2ab21, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/e4ca75916940456daa370575086c1066] to archive 2023-04-27 03:55:01,562 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 03:55:01,564 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/1defe950142546b7acdc141d69d1f2a0 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/1defe950142546b7acdc141d69d1f2a0 2023-04-27 03:55:01,565 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/736cc4ad1c4e46ce88c61b01eba7af34 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/736cc4ad1c4e46ce88c61b01eba7af34 2023-04-27 03:55:01,566 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/249d0f4e024b48d4acdd7613bad9d86c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/249d0f4e024b48d4acdd7613bad9d86c 2023-04-27 03:55:01,568 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/a892a37d297f473bb5feeac38963586f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/a892a37d297f473bb5feeac38963586f 2023-04-27 03:55:01,569 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/1c56857fd3334b388a6e075e21a2a624 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/1c56857fd3334b388a6e075e21a2a624 2023-04-27 03:55:01,570 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/bb9da04d8b684408b943cdcb7e5c77cd to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/bb9da04d8b684408b943cdcb7e5c77cd 2023-04-27 03:55:01,571 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/4ee12e19ebb34b4ab45b7e932ff97f26 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/4ee12e19ebb34b4ab45b7e932ff97f26 2023-04-27 03:55:01,572 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/93722a2a9be5466cb3425a7ad982983f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/93722a2a9be5466cb3425a7ad982983f 2023-04-27 03:55:01,573 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/301cb37987bc4d708d855775b634ea1d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/301cb37987bc4d708d855775b634ea1d 2023-04-27 03:55:01,574 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/c2e96bb6722b473e9a3627648c8c2370 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/c2e96bb6722b473e9a3627648c8c2370 2023-04-27 03:55:01,576 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/d70e4141fb61492f934ab6df75dcdfb6 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/d70e4141fb61492f934ab6df75dcdfb6 2023-04-27 03:55:01,577 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/9607be49a68246bfb48d22ee71e0de96 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/9607be49a68246bfb48d22ee71e0de96 2023-04-27 03:55:01,578 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/6ab831af0bba4d2ab74cae0d68630b7a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/6ab831af0bba4d2ab74cae0d68630b7a 2023-04-27 03:55:01,579 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/ead45b63cc24402e8e914283c4e6f387 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/ead45b63cc24402e8e914283c4e6f387 2023-04-27 03:55:01,580 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/5ac1f73a478c439f95381e70e18a8a2b to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/5ac1f73a478c439f95381e70e18a8a2b 2023-04-27 03:55:01,581 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/a50e26ef347f49f1810f9b2cc657ca71 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/a50e26ef347f49f1810f9b2cc657ca71 2023-04-27 03:55:01,581 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/52587078d9c244c28f2da9695e910002 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/52587078d9c244c28f2da9695e910002 2023-04-27 03:55:01,583 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/f5368c35d39245d98412916eda6146c8 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/f5368c35d39245d98412916eda6146c8 2023-04-27 03:55:01,584 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/07f133281b5744e09c2856ff1dd71d7e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/07f133281b5744e09c2856ff1dd71d7e 2023-04-27 03:55:01,584 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/44b8721facbc40c1a59e8f1a2c4f9ba2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/44b8721facbc40c1a59e8f1a2c4f9ba2 2023-04-27 03:55:01,585 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/d4c8ca7859e64ae1a92e6d22abcd5f5a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/d4c8ca7859e64ae1a92e6d22abcd5f5a 2023-04-27 03:55:01,586 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/8845cb603fd148a5b3efc527d8544388 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/8845cb603fd148a5b3efc527d8544388 2023-04-27 03:55:01,587 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/d124f23070bc430abbb6fa68c91a761a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/d124f23070bc430abbb6fa68c91a761a 2023-04-27 03:55:01,587 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/ccd26d3eb8a548baa245614f4027e55f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/ccd26d3eb8a548baa245614f4027e55f 2023-04-27 03:55:01,589 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/c63f23ca16d24b53a51e051021398738 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/c63f23ca16d24b53a51e051021398738 2023-04-27 03:55:01,590 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/3ad38f20722e4ac1b7d826e5173730fa to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/3ad38f20722e4ac1b7d826e5173730fa 2023-04-27 03:55:01,590 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/eadc85396c71421f8abdb2076f3b7ecc to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/eadc85396c71421f8abdb2076f3b7ecc 2023-04-27 03:55:01,591 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/d2b17b9a768445cfb7bfd397d3867ebf to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/d2b17b9a768445cfb7bfd397d3867ebf 2023-04-27 03:55:01,592 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/21ec161f247a4c5894fc791752d2ab21 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/21ec161f247a4c5894fc791752d2ab21 2023-04-27 03:55:01,593 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/e4ca75916940456daa370575086c1066 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/e4ca75916940456daa370575086c1066 2023-04-27 03:55:01,596 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/4dedea72d0764e3ea1b91fd8cf436c4e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/488315f1704f430cb5bc47fc04526c90, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/f4b63fb7cdca454791d6eb3159bc844d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/46ced774acbb4d5c8c6d81e1f35fdcb0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/fe4909d3acce4e1ba695fd3975f508bb, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/144e4278319d42b88af5042aab61a92f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/503fd21105694441a9ca1e0d978a29ea, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/4e49b5ea8cc142e3a0c83fd3c6931c28, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/53c7eba76aca4be6b6114d5969c107dd, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/5e6c28db978342d798ca095e44cee049, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/611d1ca178ea4823a9eb680bf06e4de8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/352e55b79d3b489ea37ce6f3d5e47894, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/dab540ded83c4dc5aa5207faf197069f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/9ecdb14462f04e73968bf27d326581d3, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/0cbc82a5b4294ba68157c0676bfc02d5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/2c877e982f0041499903a9cd31fd27c2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/959807de1c90480c9c48f0aff3f7d73f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/3ce2f9d5686a41c59e7a3a317ab155be, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/c40e7b3ebc2a4f93a4a1797ee0617c61, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/980b6e3ee61742ea9bebf199322ec608, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/1ed6d0622217417385d7d348279ca21d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/6fd3ee6c5fc0444d949ee036415acb9b, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/d64f78e5d190496d9746b27e59e7a50b, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/61934ef7a6d8418abcf8947210fc8bee, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/d341ddfb26ba4984b95f89be1d511566, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/6f84409d5e9441cb855d28a0f7a4e123, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/9e94dc62e3f448ad9b4b33b49a172182, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/b4403d88c2c44d328a645772bab9fafa, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/a520c0752d174bff8dc2411b4605ff49, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/f0bce6ca8b464defaa673bd7d0cf4710] to archive 2023-04-27 03:55:01,597 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 03:55:01,598 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/4dedea72d0764e3ea1b91fd8cf436c4e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/4dedea72d0764e3ea1b91fd8cf436c4e 2023-04-27 03:55:01,599 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/488315f1704f430cb5bc47fc04526c90 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/488315f1704f430cb5bc47fc04526c90 2023-04-27 03:55:01,599 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/f4b63fb7cdca454791d6eb3159bc844d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/f4b63fb7cdca454791d6eb3159bc844d 2023-04-27 03:55:01,600 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/46ced774acbb4d5c8c6d81e1f35fdcb0 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/46ced774acbb4d5c8c6d81e1f35fdcb0 2023-04-27 03:55:01,601 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/fe4909d3acce4e1ba695fd3975f508bb to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/fe4909d3acce4e1ba695fd3975f508bb 2023-04-27 03:55:01,602 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/144e4278319d42b88af5042aab61a92f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/144e4278319d42b88af5042aab61a92f 2023-04-27 03:55:01,603 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/503fd21105694441a9ca1e0d978a29ea to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/503fd21105694441a9ca1e0d978a29ea 2023-04-27 03:55:01,604 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/4e49b5ea8cc142e3a0c83fd3c6931c28 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/4e49b5ea8cc142e3a0c83fd3c6931c28 2023-04-27 03:55:01,604 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/53c7eba76aca4be6b6114d5969c107dd to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/53c7eba76aca4be6b6114d5969c107dd 2023-04-27 03:55:01,605 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/5e6c28db978342d798ca095e44cee049 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/5e6c28db978342d798ca095e44cee049 2023-04-27 03:55:01,606 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/611d1ca178ea4823a9eb680bf06e4de8 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/611d1ca178ea4823a9eb680bf06e4de8 2023-04-27 03:55:01,606 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/352e55b79d3b489ea37ce6f3d5e47894 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/352e55b79d3b489ea37ce6f3d5e47894 2023-04-27 03:55:01,607 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/dab540ded83c4dc5aa5207faf197069f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/dab540ded83c4dc5aa5207faf197069f 2023-04-27 03:55:01,608 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/9ecdb14462f04e73968bf27d326581d3 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/9ecdb14462f04e73968bf27d326581d3 2023-04-27 03:55:01,609 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/0cbc82a5b4294ba68157c0676bfc02d5 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/0cbc82a5b4294ba68157c0676bfc02d5 2023-04-27 03:55:01,610 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/2c877e982f0041499903a9cd31fd27c2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/2c877e982f0041499903a9cd31fd27c2 2023-04-27 03:55:01,610 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/959807de1c90480c9c48f0aff3f7d73f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/959807de1c90480c9c48f0aff3f7d73f 2023-04-27 03:55:01,611 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/3ce2f9d5686a41c59e7a3a317ab155be to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/3ce2f9d5686a41c59e7a3a317ab155be 2023-04-27 03:55:01,612 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/c40e7b3ebc2a4f93a4a1797ee0617c61 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/c40e7b3ebc2a4f93a4a1797ee0617c61 2023-04-27 03:55:01,613 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/980b6e3ee61742ea9bebf199322ec608 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/980b6e3ee61742ea9bebf199322ec608 2023-04-27 03:55:01,613 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/1ed6d0622217417385d7d348279ca21d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/1ed6d0622217417385d7d348279ca21d 2023-04-27 03:55:01,614 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/6fd3ee6c5fc0444d949ee036415acb9b to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/6fd3ee6c5fc0444d949ee036415acb9b 2023-04-27 03:55:01,615 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/d64f78e5d190496d9746b27e59e7a50b to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/d64f78e5d190496d9746b27e59e7a50b 2023-04-27 03:55:01,616 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/61934ef7a6d8418abcf8947210fc8bee to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/61934ef7a6d8418abcf8947210fc8bee 2023-04-27 03:55:01,616 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/d341ddfb26ba4984b95f89be1d511566 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/d341ddfb26ba4984b95f89be1d511566 2023-04-27 03:55:01,617 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/6f84409d5e9441cb855d28a0f7a4e123 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/6f84409d5e9441cb855d28a0f7a4e123 2023-04-27 03:55:01,618 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/9e94dc62e3f448ad9b4b33b49a172182 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/9e94dc62e3f448ad9b4b33b49a172182 2023-04-27 03:55:01,619 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/b4403d88c2c44d328a645772bab9fafa to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/b4403d88c2c44d328a645772bab9fafa 2023-04-27 03:55:01,620 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/a520c0752d174bff8dc2411b4605ff49 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/a520c0752d174bff8dc2411b4605ff49 2023-04-27 03:55:01,620 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/f0bce6ca8b464defaa673bd7d0cf4710 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/f0bce6ca8b464defaa673bd7d0cf4710 2023-04-27 03:55:01,623 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/a697c54514184d41baf50ee9f1271b13, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/50ec9bc5727c42d3b0a713c28adf09b9, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/53782d67c7cf47ef8e0adde64d6d129e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/e4630e195f424a30a4b8d6298320d6f5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/ac74401ef24748218ee7b68344137a75, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/27a6125fd7d9416596dc8e22fb587295, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/6cb3ee473c4144e580f8d546b01700bf, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/70ed89c647bb465da4d5b7060533571f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/bff8cec9fa984e4b8242d74e07ccedc1, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/78b300159c6c435f913fe244b6c7756e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/c6d1c350afd042dbae0fdb443c1e6228, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/a79142253d9349598bc93edd7c429794, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/294e63c233124474a5ed156440dc3ba1, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/5c8836737685436f8354736f3911418a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/272ff850ed994a6690a23be09a31f269, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/45a43f0c7d0b48e0ae25cbd999e8ced7, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/23b2500bfcbe420897251ccd50fed5d5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/5ebdabb24bd542f694bf432fbf59717d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/13b91643d0234cb784596e4cecf403d0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/98bfbad09926456386e74811381c415e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/f6e7b6803d9345a0b5200b922d16fa33, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/0c9cd4f3e7904f39ba10f59fb1cc6605, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/04a66e0639e14d329a583d5187d6b41d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/f6fb8587922e4dc49aef71e3e284fffe, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/f23a2ba3791048a68be91ce24efe1b71, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/ed42f2362ed44bfcb9586c15ec9a2da9, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/e196efe8bca34d9391364c9a8f79b8fa, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/5b85a5cdc37e466e885113a2960c3064, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/27170bd262034d88ae3497fc542a851f] to archive 2023-04-27 03:55:01,624 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 03:55:01,625 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/a697c54514184d41baf50ee9f1271b13 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/a697c54514184d41baf50ee9f1271b13 2023-04-27 03:55:01,626 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/50ec9bc5727c42d3b0a713c28adf09b9 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/50ec9bc5727c42d3b0a713c28adf09b9 2023-04-27 03:55:01,627 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/53782d67c7cf47ef8e0adde64d6d129e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/53782d67c7cf47ef8e0adde64d6d129e 2023-04-27 03:55:01,627 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/e4630e195f424a30a4b8d6298320d6f5 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/e4630e195f424a30a4b8d6298320d6f5 2023-04-27 03:55:01,628 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/ac74401ef24748218ee7b68344137a75 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/ac74401ef24748218ee7b68344137a75 2023-04-27 03:55:01,629 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/27a6125fd7d9416596dc8e22fb587295 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/27a6125fd7d9416596dc8e22fb587295 2023-04-27 03:55:01,630 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/6cb3ee473c4144e580f8d546b01700bf to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/6cb3ee473c4144e580f8d546b01700bf 2023-04-27 03:55:01,631 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/70ed89c647bb465da4d5b7060533571f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/70ed89c647bb465da4d5b7060533571f 2023-04-27 03:55:01,631 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/bff8cec9fa984e4b8242d74e07ccedc1 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/bff8cec9fa984e4b8242d74e07ccedc1 2023-04-27 03:55:01,632 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/78b300159c6c435f913fe244b6c7756e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/78b300159c6c435f913fe244b6c7756e 2023-04-27 03:55:01,633 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/c6d1c350afd042dbae0fdb443c1e6228 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/c6d1c350afd042dbae0fdb443c1e6228 2023-04-27 03:55:01,634 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/a79142253d9349598bc93edd7c429794 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/a79142253d9349598bc93edd7c429794 2023-04-27 03:55:01,635 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/294e63c233124474a5ed156440dc3ba1 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/294e63c233124474a5ed156440dc3ba1 2023-04-27 03:55:01,636 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/5c8836737685436f8354736f3911418a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/5c8836737685436f8354736f3911418a 2023-04-27 03:55:01,637 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/272ff850ed994a6690a23be09a31f269 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/272ff850ed994a6690a23be09a31f269 2023-04-27 03:55:01,638 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/45a43f0c7d0b48e0ae25cbd999e8ced7 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/45a43f0c7d0b48e0ae25cbd999e8ced7 2023-04-27 03:55:01,639 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/23b2500bfcbe420897251ccd50fed5d5 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/23b2500bfcbe420897251ccd50fed5d5 2023-04-27 03:55:01,641 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/5ebdabb24bd542f694bf432fbf59717d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/5ebdabb24bd542f694bf432fbf59717d 2023-04-27 03:55:01,642 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/13b91643d0234cb784596e4cecf403d0 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/13b91643d0234cb784596e4cecf403d0 2023-04-27 03:55:01,643 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/98bfbad09926456386e74811381c415e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/98bfbad09926456386e74811381c415e 2023-04-27 03:55:01,644 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/f6e7b6803d9345a0b5200b922d16fa33 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/f6e7b6803d9345a0b5200b922d16fa33 2023-04-27 03:55:01,645 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/0c9cd4f3e7904f39ba10f59fb1cc6605 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/0c9cd4f3e7904f39ba10f59fb1cc6605 2023-04-27 03:55:01,645 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/04a66e0639e14d329a583d5187d6b41d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/04a66e0639e14d329a583d5187d6b41d 2023-04-27 03:55:01,646 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/f6fb8587922e4dc49aef71e3e284fffe to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/f6fb8587922e4dc49aef71e3e284fffe 2023-04-27 03:55:01,647 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/f23a2ba3791048a68be91ce24efe1b71 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/f23a2ba3791048a68be91ce24efe1b71 2023-04-27 03:55:01,648 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/ed42f2362ed44bfcb9586c15ec9a2da9 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/ed42f2362ed44bfcb9586c15ec9a2da9 2023-04-27 03:55:01,649 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/e196efe8bca34d9391364c9a8f79b8fa to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/e196efe8bca34d9391364c9a8f79b8fa 2023-04-27 03:55:01,650 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/5b85a5cdc37e466e885113a2960c3064 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/5b85a5cdc37e466e885113a2960c3064 2023-04-27 03:55:01,650 DEBUG [StoreCloser-TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/27170bd262034d88ae3497fc542a851f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/27170bd262034d88ae3497fc542a851f 2023-04-27 03:55:01,654 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/recovered.edits/480.seqid, newMaxSeqId=480, maxSeqId=1 2023-04-27 03:55:01,655 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7. 2023-04-27 03:55:01,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e07ddca27cd19d46379b6939f22f78c7: 2023-04-27 03:55:01,657 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:55:01,657 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=e07ddca27cd19d46379b6939f22f78c7, regionState=CLOSED 2023-04-27 03:55:01,657 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567701657"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682567701657"}]},"ts":"1682567701657"} 2023-04-27 03:55:01,659 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-27 03:55:01,659 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure e07ddca27cd19d46379b6939f22f78c7, server=jenkins-hbase4.apache.org,40885,1682567618305 in 1.7880 sec 2023-04-27 03:55:01,660 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-27 03:55:01,660 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e07ddca27cd19d46379b6939f22f78c7, UNASSIGN in 1.7920 sec 2023-04-27 03:55:01,661 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567701660"}]},"ts":"1682567701660"} 2023-04-27 03:55:01,661 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-27 03:55:01,663 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-27 03:55:01,664 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8030 sec 2023-04-27 03:55:01,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-27 03:55:01,968 INFO [Listener at localhost/33933] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-27 03:55:01,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-27 03:55:01,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:55:01,970 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:55:01,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-27 03:55:01,971 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:55:01,976 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:55:01,977 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A, FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B, FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C, FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/recovered.edits] 2023-04-27 03:55:01,980 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/76bb0d304def43959e3964d249b260a3 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/76bb0d304def43959e3964d249b260a3 2023-04-27 03:55:01,981 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/85d845842ef044298289a1dcadeea301 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/85d845842ef044298289a1dcadeea301 2023-04-27 03:55:01,982 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/e01d2121e40a425d912b4bcd214497e6 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/e01d2121e40a425d912b4bcd214497e6 2023-04-27 03:55:01,983 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/e176ebe95f6540e3957222d93e620a5a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/A/e176ebe95f6540e3957222d93e620a5a 2023-04-27 03:55:01,984 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/0579550a47ba4b7785e1f8effb55e156 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/0579550a47ba4b7785e1f8effb55e156 2023-04-27 03:55:01,986 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/1edf0e6218114d89a5fb5a168fb4bcd4 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/1edf0e6218114d89a5fb5a168fb4bcd4 2023-04-27 03:55:01,987 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/6778a84c5f29401dba0efcb7c5089981 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/6778a84c5f29401dba0efcb7c5089981 2023-04-27 03:55:01,988 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/81c2c7b82462431abd8d060e885953c3 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/B/81c2c7b82462431abd8d060e885953c3 2023-04-27 03:55:01,990 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/32b2e56a52d04a898513f3e653814fd5 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/32b2e56a52d04a898513f3e653814fd5 2023-04-27 03:55:01,991 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/43f3667ae097434883dade6498ae29c9 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/43f3667ae097434883dade6498ae29c9 2023-04-27 03:55:01,991 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/98edde672eba4a1e8d4baebdee4acbb6 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/98edde672eba4a1e8d4baebdee4acbb6 2023-04-27 03:55:01,992 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/c5b176f815cc4c8c843f61e45b40f036 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/C/c5b176f815cc4c8c843f61e45b40f036 2023-04-27 03:55:01,995 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/recovered.edits/480.seqid to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7/recovered.edits/480.seqid 2023-04-27 03:55:01,995 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e07ddca27cd19d46379b6939f22f78c7 2023-04-27 03:55:01,995 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-27 03:55:01,997 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:55:01,998 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-27 03:55:02,000 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-27 03:55:02,000 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:55:02,000 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-27 03:55:02,001 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682567702000"}]},"ts":"9223372036854775807"} 2023-04-27 03:55:02,002 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-27 03:55:02,002 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => e07ddca27cd19d46379b6939f22f78c7, NAME => 'TestAcidGuarantees,,1682567677201.e07ddca27cd19d46379b6939f22f78c7.', STARTKEY => '', ENDKEY => ''}] 2023-04-27 03:55:02,002 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-27 03:55:02,002 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682567702002"}]},"ts":"9223372036854775807"} 2023-04-27 03:55:02,003 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-27 03:55:02,005 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:55:02,006 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 37 msec 2023-04-27 03:55:02,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-27 03:55:02,072 INFO [Listener at localhost/33933] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-27 03:55:02,083 INFO [Listener at localhost/33933] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=230 (was 235), OpenFileDescriptor=555 (was 578), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=346 (was 366), ProcessCount=172 (was 170) - ProcessCount LEAK? -, AvailableMemoryMB=11955 (was 11989) 2023-04-27 03:55:02,094 INFO [Listener at localhost/33933] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=230, OpenFileDescriptor=555, MaxFileDescriptor=60000, SystemLoadAverage=346, ProcessCount=172, AvailableMemoryMB=11955 2023-04-27 03:55:02,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-27 03:55:02,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-27 03:55:02,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-27 03:55:02,098 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-27 03:55:02,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-27 03:55:02,099 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-27 03:55:02,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-27 03:55:02,100 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:02,101 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053 empty. 2023-04-27 03:55:02,101 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:02,101 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-27 03:55:02,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-27 03:55:02,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-27 03:55:02,508 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-27 03:55:02,509 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => e0314b11a1068d19a1052690701d1053, NAME => 'TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp 2023-04-27 03:55:02,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-27 03:55:02,915 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 03:55:02,915 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing e0314b11a1068d19a1052690701d1053, disabling compactions & flushes 2023-04-27 03:55:02,915 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:02,915 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:02,915 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. after waiting 0 ms 2023-04-27 03:55:02,915 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:02,915 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:02,915 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:02,917 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-27 03:55:02,918 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567702917"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682567702917"}]},"ts":"1682567702917"} 2023-04-27 03:55:02,918 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-27 03:55:02,919 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-27 03:55:02,919 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567702919"}]},"ts":"1682567702919"} 2023-04-27 03:55:02,920 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-27 03:55:02,924 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e0314b11a1068d19a1052690701d1053, ASSIGN}] 2023-04-27 03:55:02,925 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e0314b11a1068d19a1052690701d1053, ASSIGN 2023-04-27 03:55:02,926 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e0314b11a1068d19a1052690701d1053, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40885,1682567618305; forceNewPlan=false, retain=false 2023-04-27 03:55:03,077 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=e0314b11a1068d19a1052690701d1053, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:03,077 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567703077"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682567703077"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682567703077"}]},"ts":"1682567703077"} 2023-04-27 03:55:03,078 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305}] 2023-04-27 03:55:03,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-27 03:55:03,233 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:03,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e0314b11a1068d19a1052690701d1053, NAME => 'TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.', STARTKEY => '', ENDKEY => ''} 2023-04-27 03:55:03,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:03,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 03:55:03,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:03,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:03,234 INFO [StoreOpener-e0314b11a1068d19a1052690701d1053-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:03,235 DEBUG [StoreOpener-e0314b11a1068d19a1052690701d1053-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A 2023-04-27 03:55:03,235 DEBUG [StoreOpener-e0314b11a1068d19a1052690701d1053-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A 2023-04-27 03:55:03,236 INFO [StoreOpener-e0314b11a1068d19a1052690701d1053-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:55:03,236 INFO [StoreOpener-e0314b11a1068d19a1052690701d1053-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e0314b11a1068d19a1052690701d1053 columnFamilyName A 2023-04-27 03:55:03,236 INFO [StoreOpener-e0314b11a1068d19a1052690701d1053-1] regionserver.HStore(310): Store=e0314b11a1068d19a1052690701d1053/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:55:03,236 INFO [StoreOpener-e0314b11a1068d19a1052690701d1053-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:03,237 DEBUG [StoreOpener-e0314b11a1068d19a1052690701d1053-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B 2023-04-27 03:55:03,237 DEBUG [StoreOpener-e0314b11a1068d19a1052690701d1053-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B 2023-04-27 03:55:03,237 INFO [StoreOpener-e0314b11a1068d19a1052690701d1053-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:55:03,238 INFO [StoreOpener-e0314b11a1068d19a1052690701d1053-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e0314b11a1068d19a1052690701d1053 columnFamilyName B 2023-04-27 03:55:03,238 INFO [StoreOpener-e0314b11a1068d19a1052690701d1053-1] regionserver.HStore(310): Store=e0314b11a1068d19a1052690701d1053/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:55:03,238 INFO [StoreOpener-e0314b11a1068d19a1052690701d1053-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:03,239 DEBUG [StoreOpener-e0314b11a1068d19a1052690701d1053-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C 2023-04-27 03:55:03,239 DEBUG [StoreOpener-e0314b11a1068d19a1052690701d1053-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C 2023-04-27 03:55:03,239 INFO [StoreOpener-e0314b11a1068d19a1052690701d1053-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:55:03,239 INFO [StoreOpener-e0314b11a1068d19a1052690701d1053-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e0314b11a1068d19a1052690701d1053 columnFamilyName C 2023-04-27 03:55:03,240 INFO [StoreOpener-e0314b11a1068d19a1052690701d1053-1] regionserver.HStore(310): Store=e0314b11a1068d19a1052690701d1053/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:55:03,240 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:03,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:03,241 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:03,242 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-27 03:55:03,243 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:03,244 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-27 03:55:03,245 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e0314b11a1068d19a1052690701d1053; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71253260, jitterRate=0.06175631284713745}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-27 03:55:03,245 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:03,245 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., pid=37, masterSystemTime=1682567703230 2023-04-27 03:55:03,247 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:03,247 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:03,247 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=e0314b11a1068d19a1052690701d1053, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:03,247 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567703247"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682567703247"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682567703247"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682567703247"}]},"ts":"1682567703247"} 2023-04-27 03:55:03,249 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-27 03:55:03,249 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 in 170 msec 2023-04-27 03:55:03,250 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-27 03:55:03,250 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e0314b11a1068d19a1052690701d1053, ASSIGN in 325 msec 2023-04-27 03:55:03,251 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-27 03:55:03,251 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567703251"}]},"ts":"1682567703251"} 2023-04-27 03:55:03,251 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-27 03:55:03,254 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-27 03:55:03,254 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1570 sec 2023-04-27 03:55:04,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-27 03:55:04,203 INFO [Listener at localhost/33933] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-27 03:55:04,204 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x0bcfc1e0 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:55:04,209 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@445919fd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:55:04,210 DEBUG [Listener at localhost/33933] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:55:04,212 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59730, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:55:04,213 DEBUG [Listener at localhost/33933] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-27 03:55:04,214 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50994, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-27 03:55:04,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-27 03:55:04,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-27 03:55:04,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-27 03:55:04,625 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-27 03:55:04,626 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-27 03:55:04,626 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-27 03:55:04,628 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-27 03:55:04,630 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e0314b11a1068d19a1052690701d1053, REOPEN/MOVE}] 2023-04-27 03:55:04,631 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e0314b11a1068d19a1052690701d1053, REOPEN/MOVE 2023-04-27 03:55:04,631 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=e0314b11a1068d19a1052690701d1053, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:04,631 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567704631"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682567704631"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682567704631"}]},"ts":"1682567704631"} 2023-04-27 03:55:04,632 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305}] 2023-04-27 03:55:04,784 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:04,785 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e0314b11a1068d19a1052690701d1053, disabling compactions & flushes 2023-04-27 03:55:04,785 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:04,785 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:04,785 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. after waiting 0 ms 2023-04-27 03:55:04,785 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:04,790 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-27 03:55:04,790 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:04,790 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:04,790 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: e0314b11a1068d19a1052690701d1053 to self. 2023-04-27 03:55:04,792 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:04,792 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=e0314b11a1068d19a1052690701d1053, regionState=CLOSED 2023-04-27 03:55:04,792 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567704792"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682567704792"}]},"ts":"1682567704792"} 2023-04-27 03:55:04,794 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-27 03:55:04,794 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 in 161 msec 2023-04-27 03:55:04,794 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e0314b11a1068d19a1052690701d1053, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,40885,1682567618305; forceNewPlan=false, retain=true 2023-04-27 03:55:04,945 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=e0314b11a1068d19a1052690701d1053, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:04,945 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567704945"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682567704945"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682567704945"}]},"ts":"1682567704945"} 2023-04-27 03:55:04,946 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305}] 2023-04-27 03:55:05,100 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:05,101 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e0314b11a1068d19a1052690701d1053, NAME => 'TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.', STARTKEY => '', ENDKEY => ''} 2023-04-27 03:55:05,101 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:05,102 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 03:55:05,102 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:05,102 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:05,103 INFO [StoreOpener-e0314b11a1068d19a1052690701d1053-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:05,103 DEBUG [StoreOpener-e0314b11a1068d19a1052690701d1053-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A 2023-04-27 03:55:05,104 DEBUG [StoreOpener-e0314b11a1068d19a1052690701d1053-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A 2023-04-27 03:55:05,104 INFO [StoreOpener-e0314b11a1068d19a1052690701d1053-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:55:05,104 INFO [StoreOpener-e0314b11a1068d19a1052690701d1053-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e0314b11a1068d19a1052690701d1053 columnFamilyName A 2023-04-27 03:55:05,105 INFO [StoreOpener-e0314b11a1068d19a1052690701d1053-1] regionserver.HStore(310): Store=e0314b11a1068d19a1052690701d1053/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:55:05,105 INFO [StoreOpener-e0314b11a1068d19a1052690701d1053-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:05,106 DEBUG [StoreOpener-e0314b11a1068d19a1052690701d1053-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B 2023-04-27 03:55:05,106 DEBUG [StoreOpener-e0314b11a1068d19a1052690701d1053-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B 2023-04-27 03:55:05,106 INFO [StoreOpener-e0314b11a1068d19a1052690701d1053-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:55:05,107 INFO [StoreOpener-e0314b11a1068d19a1052690701d1053-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e0314b11a1068d19a1052690701d1053 columnFamilyName B 2023-04-27 03:55:05,107 INFO [StoreOpener-e0314b11a1068d19a1052690701d1053-1] regionserver.HStore(310): Store=e0314b11a1068d19a1052690701d1053/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:55:05,107 INFO [StoreOpener-e0314b11a1068d19a1052690701d1053-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:05,108 DEBUG [StoreOpener-e0314b11a1068d19a1052690701d1053-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C 2023-04-27 03:55:05,108 DEBUG [StoreOpener-e0314b11a1068d19a1052690701d1053-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C 2023-04-27 03:55:05,108 INFO [StoreOpener-e0314b11a1068d19a1052690701d1053-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:55:05,108 INFO [StoreOpener-e0314b11a1068d19a1052690701d1053-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e0314b11a1068d19a1052690701d1053 columnFamilyName C 2023-04-27 03:55:05,108 INFO [StoreOpener-e0314b11a1068d19a1052690701d1053-1] regionserver.HStore(310): Store=e0314b11a1068d19a1052690701d1053/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:55:05,108 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:05,109 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:05,110 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:05,111 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-27 03:55:05,112 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:05,113 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e0314b11a1068d19a1052690701d1053; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60139213, jitterRate=-0.10385589301586151}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-27 03:55:05,113 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:05,113 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., pid=42, masterSystemTime=1682567705098 2023-04-27 03:55:05,114 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:05,114 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:05,115 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=e0314b11a1068d19a1052690701d1053, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,115 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567705115"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682567705115"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682567705115"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682567705115"}]},"ts":"1682567705115"} 2023-04-27 03:55:05,116 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-27 03:55:05,117 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 in 170 msec 2023-04-27 03:55:05,117 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-27 03:55:05,118 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e0314b11a1068d19a1052690701d1053, REOPEN/MOVE in 487 msec 2023-04-27 03:55:05,119 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-27 03:55:05,119 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 490 msec 2023-04-27 03:55:05,120 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 904 msec 2023-04-27 03:55:05,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-27 03:55:05,122 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x38341134 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:55:05,126 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d010713, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:55:05,127 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x71b5058b to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:55:05,131 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69100738, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:55:05,132 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x3620ee35 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:55:05,135 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f603c27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:55:05,135 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x4fe1db6a to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:55:05,139 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e01d5e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:55:05,139 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x10377b50 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:55:05,142 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51ccd8b7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:55:05,142 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x68a0e21a to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:55:05,146 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9e86d9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:55:05,147 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x1ec12b02 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:55:05,150 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@711ebaf2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:55:05,150 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x1d7ba242 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:55:05,155 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a7503d4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:55:05,155 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x3e079da1 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:55:05,158 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7eae4191, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:55:05,159 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x159c5e0c to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:55:05,163 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1774338b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:55:05,169 DEBUG [hconnection-0x5f0bf7e8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:55:05,169 DEBUG [hconnection-0x2d530d52-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:55:05,169 DEBUG [hconnection-0xabed19e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:55:05,170 DEBUG [hconnection-0x3c5e88b1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:55:05,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:05,171 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59732, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:55:05,171 DEBUG [hconnection-0x53e2c6a5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:55:05,171 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59756, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:55:05,171 DEBUG [hconnection-0x7efbe3b3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:55:05,173 DEBUG [hconnection-0x69397925-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:55:05,174 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59744, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:55:05,174 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59742, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:55:05,173 DEBUG [hconnection-0x3d622a87-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:55:05,175 DEBUG [hconnection-0x3c9c9568-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:55:05,175 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59784, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:55:05,177 DEBUG [hconnection-0x5431b8e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:55:05,179 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59768, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:55:05,179 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59762, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:55:05,179 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59802, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:55:05,180 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59798, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:55:05,180 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59812, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:55:05,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:05,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:05,183 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:05,183 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:05,183 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:05,183 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:05,185 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,186 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,187 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:05,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:05,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:05,188 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:05,190 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,190 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:05,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:05,192 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e0314b11a1068d19a1052690701d1053 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-27 03:55:05,193 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:05,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:05,193 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:05,193 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:05,194 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:05,194 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=A 2023-04-27 03:55:05,194 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:05,194 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:05,194 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:05,194 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:05,195 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:05,194 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=B 2023-04-27 03:55:05,195 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:05,195 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:05,195 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:05,195 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=C 2023-04-27 03:55:05,195 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:05,195 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:05,195 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:05,195 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,195 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:05,195 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:05,200 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,201 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,201 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:05,204 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:05,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:05,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:05,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:05,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:05,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:05,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d01363a[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:05,206 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:05,206 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,207 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,207 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:05,207 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:05,207 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:05,209 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:05,210 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:05,210 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:05,210 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,210 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:05,210 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:05,210 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:05,210 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:05,210 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:05,211 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:05,211 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:05,213 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,213 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,213 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:05,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:05,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:05,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:05,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:05,213 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:05,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:05,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,215 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:05,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,217 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:05,217 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:05,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567765210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:05,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:05,217 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:05,217 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:05,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:05,217 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,217 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:05,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:05,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:05,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:05,217 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:05,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:05,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59744 deadline: 1682567765211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,217 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,218 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,218 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,218 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,218 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:05,218 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:05,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567765212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,218 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:05,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567765217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:05,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567765218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,318 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:05,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:05,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:05,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:05,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567765318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:05,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59744 deadline: 1682567765319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:05,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567765319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:05,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567765320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:05,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567765321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:05,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:05,327 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:05,327 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:05,327 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:05,327 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:05,327 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,327 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,328 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:05,328 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:05,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:05,329 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,329 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:05,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:05,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:05,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:05,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:05,336 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:05,336 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:05,336 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:05,336 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:05,336 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:05,336 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:05,336 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:05,336 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:05,336 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:05,336 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:05,336 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:05,336 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,336 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:05,336 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:05,340 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,340 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:05,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:05,341 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,341 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:05,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:05,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:05,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:05,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:05,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d768310[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:05,342 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:05,342 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,343 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,344 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:05,344 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:05,344 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:05,344 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,344 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:05,345 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,345 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:05,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:05,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:05,345 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:05,345 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:05,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:05,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:05,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:05,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:05,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:05,347 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:05,347 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:05,347 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:05,347 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:05,347 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:05,349 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,349 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:05,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:05,349 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:05,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:05,349 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:05,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:05,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:05,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,351 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:05,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,354 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,354 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:05,354 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:05,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:05,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:05,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:05,354 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:05,354 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,354 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,354 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:05,354 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,354 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:05,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:05,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:05,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:05,354 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:05,354 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,355 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,455 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:05,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:05,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:05,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:05,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:05,458 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:05,458 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:05,458 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:05,458 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:05,459 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,459 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,460 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:05,460 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:05,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:05,461 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,461 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:05,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:05,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:05,463 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:05,463 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:05,463 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:05,463 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:05,463 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:05,463 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:05,463 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:05,463 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:05,463 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:05,463 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:05,464 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:05,464 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:05,464 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:05,464 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,464 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:05,464 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:05,466 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:05,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:05,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:05,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:05,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:05,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:05,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:05,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40954675[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:05,468 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:05,468 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,469 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,469 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:05,469 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:05,469 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:05,470 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,470 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:05,472 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:05,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:05,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:05,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:05,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:05,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:05,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:05,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:05,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:05,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:05,474 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:05,474 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:05,474 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:05,474 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:05,474 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:05,476 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,476 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:05,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:05,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,476 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:05,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:05,476 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:05,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:05,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:05,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,478 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:05,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,481 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,481 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,481 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:05,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:05,481 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:05,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:05,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:05,481 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:05,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:05,481 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:05,481 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,482 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:05,482 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:05,482 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:05,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:05,482 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,482 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:05,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567765523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:05,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567765523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:05,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59744 deadline: 1682567765524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:05,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567765527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:05,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567765527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,582 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:05,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:05,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:05,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:05,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:05,587 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:05,587 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:05,587 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:05,587 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:05,592 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,592 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,593 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,593 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:05,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:05,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:05,593 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,593 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:05,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:05,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:05,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:05,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:05,595 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:05,595 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:05,595 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:05,595 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:05,595 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:05,595 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:05,595 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:05,595 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:05,595 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:05,595 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:05,595 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:05,595 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,596 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:05,596 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:05,598 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:05,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:05,598 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,598 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:05,598 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:05,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:05,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:05,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:05,599 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:05,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56586b0d[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:05,599 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,601 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:05,601 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:05,601 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:05,601 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:05,602 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:05,603 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:05,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:05,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:05,603 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:05,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:05,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:05,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:05,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:05,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:05,605 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:05,605 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:05,605 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:05,605 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:05,605 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:05,607 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,607 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,607 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:05,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:05,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:05,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:05,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:05,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:05,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,609 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:05,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,610 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:05,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,614 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,614 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,614 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,614 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:05,614 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,614 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:05,614 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:05,614 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:05,614 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,615 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:05,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:05,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:05,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:05,615 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:05,615 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:05,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:05,629 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427356830f9ba1449378c7fd90c43401aec to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427356830f9ba1449378c7fd90c43401aec 2023-04-27 03:55:05,715 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:05,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:05,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:05,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:05,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:05,720 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:05,720 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:05,720 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:05,720 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:05,721 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,721 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,722 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,722 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:05,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:05,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:05,722 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,722 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:05,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:05,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,723 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:05,723 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:05,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:05,724 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:05,724 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:05,724 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:05,724 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:05,724 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:05,724 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:05,724 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:05,724 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:05,724 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:05,724 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:05,724 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:05,724 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,724 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:05,724 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:05,726 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,726 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:05,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:05,726 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,726 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:05,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:05,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:05,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:05,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:05,728 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:05,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72cc271a[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:05,728 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,729 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:05,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:05,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:05,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:05,734 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,734 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:05,734 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:05,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:05,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:05,734 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:05,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:05,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:05,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:05,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:05,736 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:05,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:05,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:05,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:05,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:05,736 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:05,738 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,738 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:05,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,738 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:05,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:05,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:05,739 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:05,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,740 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:05,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,742 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,742 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:05,742 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:05,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:05,742 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:05,742 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:05,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:05,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:05,742 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,742 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,743 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,743 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,743 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:05,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:05,743 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:05,743 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:05,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:05,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567765829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:05,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567765829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:05,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567765830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:05,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59744 deadline: 1682567765830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:05,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567765832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,843 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:05,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:05,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:05,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:05,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:05,848 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:05,848 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:05,848 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:05,848 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:05,849 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,849 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,850 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,850 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:05,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:05,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:05,850 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,850 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:05,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:05,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:05,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:05,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:05,852 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:05,852 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:05,852 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:05,853 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:05,853 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:05,853 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:05,854 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:05,854 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:05,854 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:05,854 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:05,854 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:05,854 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,854 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:05,854 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:05,856 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:05,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:05,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:05,857 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,857 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:05,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:05,858 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:05,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b1b5450[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:05,858 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,859 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,859 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:05,859 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:05,860 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:05,860 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:05,861 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:05,861 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:05,861 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,861 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:05,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:05,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:05,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:05,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:05,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:05,864 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:05,864 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:05,864 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:05,864 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:05,865 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:05,866 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,866 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:05,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:05,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,866 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:05,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:05,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:05,868 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:05,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,869 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:05,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,873 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:05,873 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:05,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:05,873 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:05,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:05,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:05,873 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,873 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,873 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,873 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,873 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,873 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,873 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:05,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:05,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:05,873 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:05,873 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:05,974 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:05,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:05,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:05,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:05,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:05,976 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:05,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:05,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:05,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:05,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,978 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:05,978 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:05,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:05,979 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,979 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:05,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:05,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:05,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:05,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:05,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:05,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:05,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:05,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:05,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:05,980 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:05,980 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:05,980 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:05,980 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:05,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:05,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:05,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:05,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:05,982 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:05,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:05,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:05,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:05,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:05,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:05,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:05,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:05,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@215715a8[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:05,985 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:05,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:05,985 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,985 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:05,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:05,989 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:05,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:05,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:05,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:05,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:05,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:05,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:05,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:05,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:05,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:05,991 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:05,992 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:05,992 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:05,992 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:05,992 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:05,993 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:05,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:05,994 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,994 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,994 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:05,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:05,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:05,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:05,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,995 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:05,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:05,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,997 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,997 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:05,997 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:05,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:05,997 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,997 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:05,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:05,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:05,997 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:05,997 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:05,997 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,997 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:05,997 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:05,997 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:05,998 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:05,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:05,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:05,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:06,034 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=18, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/1b46db9689f14125a42575828a9a5f69 2023-04-27 03:55:06,043 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/5c19e4af3dc147498eab4edfc8f69114 2023-04-27 03:55:06,054 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/8ba218714d5144f7be0831b85d3e2214 2023-04-27 03:55:06,057 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/1b46db9689f14125a42575828a9a5f69 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/1b46db9689f14125a42575828a9a5f69 2023-04-27 03:55:06,062 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/1b46db9689f14125a42575828a9a5f69, entries=150, sequenceid=18, filesize=25.1 K 2023-04-27 03:55:06,063 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/5c19e4af3dc147498eab4edfc8f69114 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/5c19e4af3dc147498eab4edfc8f69114 2023-04-27 03:55:06,066 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/5c19e4af3dc147498eab4edfc8f69114, entries=150, sequenceid=18, filesize=11.6 K 2023-04-27 03:55:06,066 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/8ba218714d5144f7be0831b85d3e2214 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/8ba218714d5144f7be0831b85d3e2214 2023-04-27 03:55:06,069 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/8ba218714d5144f7be0831b85d3e2214, entries=150, sequenceid=18, filesize=11.6 K 2023-04-27 03:55:06,070 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for e0314b11a1068d19a1052690701d1053 in 878ms, sequenceid=18, compaction requested=false 2023-04-27 03:55:06,070 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:06,098 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:06,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:06,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:06,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:06,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:06,101 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:06,101 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:06,101 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:06,101 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:06,101 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,101 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:06,103 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:06,103 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:06,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:06,103 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,103 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:06,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:06,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:06,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:06,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:06,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:06,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:06,112 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:06,112 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:06,112 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:06,116 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:06,116 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:06,116 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:06,116 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:06,116 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e0314b11a1068d19a1052690701d1053 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-27 03:55:06,116 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=A 2023-04-27 03:55:06,116 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:06,116 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=B 2023-04-27 03:55:06,117 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:06,117 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=C 2023-04-27 03:55:06,117 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:06,139 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304275fc765eab8274000b8275a36be559aa6 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304275fc765eab8274000b8275a36be559aa6 2023-04-27 03:55:06,144 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/285abf99c1244b518792bda18c4e7497 2023-04-27 03:55:06,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:06,335 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:06,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:06,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59744 deadline: 1682567766345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:06,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567766346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:06,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567766348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:06,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567766349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:06,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567766349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,451 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:06,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59744 deadline: 1682567766450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:06,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567766450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:06,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567766454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:06,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567766454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:06,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567766455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,554 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/d0c769c48a19435782ba3202fe68d5c4 2023-04-27 03:55:06,564 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/d59863b993da4474bfd9326b0c36bb51 2023-04-27 03:55:06,568 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/285abf99c1244b518792bda18c4e7497 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/285abf99c1244b518792bda18c4e7497 2023-04-27 03:55:06,571 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/285abf99c1244b518792bda18c4e7497, entries=150, sequenceid=36, filesize=25.1 K 2023-04-27 03:55:06,572 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/d0c769c48a19435782ba3202fe68d5c4 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/d0c769c48a19435782ba3202fe68d5c4 2023-04-27 03:55:06,575 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/d0c769c48a19435782ba3202fe68d5c4, entries=150, sequenceid=36, filesize=11.6 K 2023-04-27 03:55:06,576 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/d59863b993da4474bfd9326b0c36bb51 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/d59863b993da4474bfd9326b0c36bb51 2023-04-27 03:55:06,579 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/d59863b993da4474bfd9326b0c36bb51, entries=150, sequenceid=36, filesize=11.6 K 2023-04-27 03:55:06,579 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for e0314b11a1068d19a1052690701d1053 in 463ms, sequenceid=36, compaction requested=false 2023-04-27 03:55:06,579 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:06,580 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:06,580 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:06,580 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:06,580 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,580 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:06,580 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:06,582 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,582 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:06,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:06,583 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,583 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:06,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:06,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:06,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:06,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:06,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:06,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3442028c[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:06,584 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:06,584 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,585 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,585 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:06,585 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:06,585 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:06,585 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,586 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:06,587 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,587 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:06,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:06,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:06,587 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:06,587 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:06,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:06,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:06,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:06,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:06,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:06,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:06,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:06,589 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:06,589 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:06,589 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:06,589 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:06,589 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:06,594 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,594 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:06,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:06,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,594 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:06,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:06,595 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:06,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:06,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:06,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:06,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:06,596 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:06,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:06,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,599 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,599 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:06,599 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:06,599 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,599 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:06,599 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:06,599 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:06,599 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,600 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,600 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:06,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:06,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:06,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:06,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:06,600 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:06,600 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:06,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:06,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:06,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:06,656 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e0314b11a1068d19a1052690701d1053 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-27 03:55:06,656 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=A 2023-04-27 03:55:06,656 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:06,656 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=B 2023-04-27 03:55:06,656 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:06,656 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=C 2023-04-27 03:55:06,656 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:06,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:06,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59744 deadline: 1682567766684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:06,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567766687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:06,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567766690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:06,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567766691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:06,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567766692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,700 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:06,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:06,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:06,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:06,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:06,703 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:06,703 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:06,703 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:06,703 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:06,704 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,704 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:06,705 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:06,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:06,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:06,706 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:06,706 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,706 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:06,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:06,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:06,707 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:06,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:06,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:06,709 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:06,709 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:06,709 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:06,710 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:06,710 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:06,714 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:06,714 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:06,714 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:06,714 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:06,714 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:06,714 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:06,714 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,715 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:06,715 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:06,716 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,716 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:06,716 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,716 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:06,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:06,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:06,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:06,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:06,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:06,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@272517e9[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:06,718 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:06,718 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,721 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:06,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:06,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:06,721 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,721 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:06,722 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:06,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:06,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:06,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:06,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:06,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:06,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:06,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:06,724 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:06,724 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:06,725 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:06,725 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:06,725 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:06,726 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,726 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:06,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:06,726 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:06,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:06,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:06,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:06,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:06,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:06,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:06,728 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:06,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:06,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,730 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,730 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:06,730 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:06,730 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:06,730 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:06,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:06,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:06,730 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:06,730 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,730 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,730 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,731 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:06,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:06,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:06,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:06,731 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:06,731 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:06,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:06,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59744 deadline: 1682567766793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:06,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567766797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:06,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567766798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:06,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567766798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:06,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567766799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,831 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:06,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:06,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:06,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:06,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:06,834 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:06,834 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:06,834 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:06,834 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:06,835 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,835 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:06,837 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:06,837 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:06,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:06,837 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,837 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:06,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:06,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:06,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:06,838 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:06,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:06,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:06,838 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:06,838 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:06,838 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:06,838 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:06,839 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:06,839 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:06,839 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:06,839 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:06,839 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:06,839 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:06,839 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:06,839 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,839 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:06,839 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:06,840 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,840 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:06,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:06,841 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,841 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:06,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:06,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:06,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:06,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:06,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:06,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@360d7e89[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:06,842 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:06,842 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,843 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,843 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:06,843 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:06,843 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:06,843 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,844 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:06,845 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:06,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:06,845 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:06,845 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:06,845 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:06,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:06,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:06,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:06,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:06,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:06,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:06,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:06,847 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:06,847 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:06,847 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:06,847 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:06,847 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:06,850 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,850 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:06,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:06,850 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:06,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:06,851 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:06,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:06,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:06,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:06,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:06,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:06,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:06,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,854 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:06,854 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:06,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:06,854 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:06,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:06,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:06,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:06,854 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:06,854 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:06,854 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:06,855 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:06,855 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,855 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:06,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:06,854 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:06,855 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,855 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,955 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:06,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:06,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:06,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:06,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:06,958 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:06,958 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:06,958 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:06,958 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:06,959 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,959 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:06,960 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:06,960 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:06,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:06,960 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,960 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:06,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:06,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:06,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:06,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:06,961 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:06,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:06,961 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:06,961 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:06,961 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:06,962 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:06,962 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:06,962 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:06,962 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:06,962 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:06,962 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:06,962 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:06,962 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:06,962 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,962 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:06,962 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:06,963 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,963 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:06,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:06,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:06,964 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,964 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:06,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:06,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:06,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:06,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:06,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2feec7b5[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:06,965 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:06,965 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,967 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,967 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:06,967 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:06,967 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:06,967 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,967 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:06,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:06,968 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:06,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:06,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:06,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:06,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:06,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:06,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:06,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:06,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:06,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:06,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:06,971 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:06,971 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:06,971 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:06,971 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:06,971 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:06,973 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,973 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:06,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:06,973 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:06,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:06,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:06,973 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:06,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:06,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:06,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:06,974 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:06,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:06,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:06,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,984 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,984 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:06,984 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:06,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:06,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:06,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:06,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:06,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:06,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:06,984 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:06,984 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:06,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:06,984 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:06,984 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:06,985 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:06,985 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:06,985 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:06,985 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:07,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59744 deadline: 1682567766999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:07,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567767003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:07,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567767003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:07,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567767003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:07,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567767004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,085 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:07,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:07,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:07,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:07,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:07,089 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:07,089 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:07,089 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:07,089 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:07,089 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,089 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,090 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427aee275071b504f6c90ea08103eea0009 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427aee275071b504f6c90ea08103eea0009 2023-04-27 03:55:07,091 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:07,091 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:07,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:07,092 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,092 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:07,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:07,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:07,094 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:07,094 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:07,094 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:07,094 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:07,094 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:07,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:07,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:07,095 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:07,095 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:07,095 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:07,095 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:07,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:07,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:07,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:07,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:07,097 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,097 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,097 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:07,098 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:07,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:07,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:07,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:07,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:07,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:07,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b1ce6a6[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:07,101 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:07,101 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,102 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,102 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:07,102 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,102 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:07,102 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:07,102 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:07,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:07,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:07,108 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:07,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:07,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:07,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:07,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:07,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:07,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:07,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:07,110 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:07,111 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:07,111 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:07,111 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:07,111 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:07,112 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,112 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:07,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:07,112 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:07,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:07,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:07,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:07,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:07,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,114 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:07,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,116 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,116 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:07,116 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:07,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:07,117 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,117 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:07,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:07,117 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:07,117 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:07,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:07,117 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:07,117 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,117 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:07,117 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:07,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:07,186 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-27 03:55:07,217 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:07,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:07,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:07,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:07,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:07,221 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:07,221 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:07,221 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:07,221 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:07,222 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,222 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,223 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,223 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:07,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:07,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:07,223 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,223 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:07,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:07,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,224 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:07,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:07,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:07,224 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:07,225 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:07,225 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:07,225 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:07,225 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:07,225 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:07,225 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:07,225 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:07,225 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:07,225 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:07,225 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:07,225 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,225 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:07,225 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:07,227 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:07,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:07,228 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:07,228 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,228 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:07,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:07,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:07,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:07,229 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:07,229 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4892273[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:07,231 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,232 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:07,232 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:07,232 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,232 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:07,232 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:07,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:07,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:07,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:07,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:07,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:07,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:07,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:07,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:07,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:07,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:07,235 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:07,235 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:07,235 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:07,235 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:07,236 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:07,237 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,237 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:07,237 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:07,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:07,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:07,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:07,238 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:07,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:07,239 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:07,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,244 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:07,244 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:07,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:07,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:07,244 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:07,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:07,244 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:07,244 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:07,244 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:07,244 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:07,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:07,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:07,244 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,245 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,245 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,245 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,245 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:07,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59744 deadline: 1682567767306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:07,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567767308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:07,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567767309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,313 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:07,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567767310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:07,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567767312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,344 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:07,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:07,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:07,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:07,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:07,348 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:07,348 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:07,348 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:07,348 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:07,348 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,348 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,350 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:07,350 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:07,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:07,350 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,350 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:07,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:07,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:07,351 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:07,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:07,351 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:07,351 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:07,351 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:07,351 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:07,351 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:07,351 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:07,351 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:07,352 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:07,352 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:07,352 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:07,352 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:07,352 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,352 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:07,352 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:07,353 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,353 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:07,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:07,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:07,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:07,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:07,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:07,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:07,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59aded5d[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:07,355 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:07,355 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,356 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,356 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:07,356 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:07,356 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:07,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:07,359 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:07,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:07,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:07,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:07,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:07,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:07,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:07,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:07,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:07,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:07,361 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:07,361 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:07,361 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:07,361 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:07,361 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:07,363 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,363 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:07,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:07,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,363 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:07,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:07,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:07,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:07,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,364 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,365 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:07,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:07,370 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:07,370 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:07,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:07,370 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:07,370 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,370 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,370 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,370 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,370 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,370 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:07,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:07,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:07,371 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:07,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:07,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:07,372 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:07,372 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:07,472 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:07,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:07,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:07,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:07,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:07,475 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:07,475 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:07,475 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:07,475 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:07,475 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,475 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,477 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:07,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:07,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:07,478 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,478 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:07,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:07,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:07,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:07,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:07,479 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:07,479 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:07,479 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:07,479 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:07,479 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:07,479 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:07,479 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:07,479 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:07,479 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:07,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:07,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:07,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:07,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:07,481 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:07,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:07,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:07,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:07,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:07,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:07,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:07,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:07,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a74ab2[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:07,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,487 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,487 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:07,487 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:07,487 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:07,487 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,487 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:07,488 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,488 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:07,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:07,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:07,489 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:07,489 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:07,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:07,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:07,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:07,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:07,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:07,491 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:07,491 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:07,491 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:07,491 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:07,491 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:07,494 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,494 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:07,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:07,494 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:07,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:07,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:07,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:07,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:07,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,497 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:07,497 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=52, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/8196600520d94233a9b04e47216cd086 2023-04-27 03:55:07,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,498 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:07,501 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:07,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:07,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:07,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:07,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:07,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:07,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,501 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:07,501 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:07,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:07,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:07,512 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/e381188d9ade43db8e0624b39e8bfccb 2023-04-27 03:55:07,523 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/614c85b0efc04e189067c3a9bd4ba3e8 2023-04-27 03:55:07,527 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/8196600520d94233a9b04e47216cd086 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/8196600520d94233a9b04e47216cd086 2023-04-27 03:55:07,530 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/8196600520d94233a9b04e47216cd086, entries=150, sequenceid=52, filesize=25.1 K 2023-04-27 03:55:07,531 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/e381188d9ade43db8e0624b39e8bfccb as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/e381188d9ade43db8e0624b39e8bfccb 2023-04-27 03:55:07,534 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/e381188d9ade43db8e0624b39e8bfccb, entries=150, sequenceid=52, filesize=11.6 K 2023-04-27 03:55:07,535 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/614c85b0efc04e189067c3a9bd4ba3e8 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/614c85b0efc04e189067c3a9bd4ba3e8 2023-04-27 03:55:07,541 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/614c85b0efc04e189067c3a9bd4ba3e8, entries=150, sequenceid=52, filesize=11.6 K 2023-04-27 03:55:07,541 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for e0314b11a1068d19a1052690701d1053 in 885ms, sequenceid=52, compaction requested=true 2023-04-27 03:55:07,542 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:07,542 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:07,542 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:07,542 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:07,542 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:07,542 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:07,543 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:07,543 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): e0314b11a1068d19a1052690701d1053/A is initiating minor compaction (all files) 2023-04-27 03:55:07,543 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:07,543 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e0314b11a1068d19a1052690701d1053/A in TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:07,543 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e0314b11a1068d19a1052690701d1053/B is initiating minor compaction (all files) 2023-04-27 03:55:07,543 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/1b46db9689f14125a42575828a9a5f69, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/285abf99c1244b518792bda18c4e7497, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/8196600520d94233a9b04e47216cd086] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp, totalSize=75.4 K 2023-04-27 03:55:07,543 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e0314b11a1068d19a1052690701d1053/B in TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:07,543 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/5c19e4af3dc147498eab4edfc8f69114, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/d0c769c48a19435782ba3202fe68d5c4, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/e381188d9ade43db8e0624b39e8bfccb] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp, totalSize=34.8 K 2023-04-27 03:55:07,543 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 1b46db9689f14125a42575828a9a5f69, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1682567705190 2023-04-27 03:55:07,543 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 5c19e4af3dc147498eab4edfc8f69114, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1682567705190 2023-04-27 03:55:07,544 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 285abf99c1244b518792bda18c4e7497, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1682567705209 2023-04-27 03:55:07,544 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting d0c769c48a19435782ba3202fe68d5c4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1682567705209 2023-04-27 03:55:07,545 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 8196600520d94233a9b04e47216cd086, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1682567706344 2023-04-27 03:55:07,545 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting e381188d9ade43db8e0624b39e8bfccb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1682567706344 2023-04-27 03:55:07,550 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e0314b11a1068d19a1052690701d1053#B#compaction#316 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:07,552 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): e0314b11a1068d19a1052690701d1053#A#compaction#315 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:07,570 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/f8aa684a3260406eb20279223e03caea as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/f8aa684a3260406eb20279223e03caea 2023-04-27 03:55:07,582 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e0314b11a1068d19a1052690701d1053/B of e0314b11a1068d19a1052690701d1053 into f8aa684a3260406eb20279223e03caea(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:07,582 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:07,582 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., storeName=e0314b11a1068d19a1052690701d1053/B, priority=13, startTime=1682567707542; duration=0sec 2023-04-27 03:55:07,582 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:07,582 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:07,583 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:07,583 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e0314b11a1068d19a1052690701d1053/C is initiating minor compaction (all files) 2023-04-27 03:55:07,583 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e0314b11a1068d19a1052690701d1053/C in TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:07,583 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/8ba218714d5144f7be0831b85d3e2214, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/d59863b993da4474bfd9326b0c36bb51, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/614c85b0efc04e189067c3a9bd4ba3e8] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp, totalSize=34.8 K 2023-04-27 03:55:07,583 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 8ba218714d5144f7be0831b85d3e2214, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1682567705190 2023-04-27 03:55:07,584 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting d59863b993da4474bfd9326b0c36bb51, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1682567705209 2023-04-27 03:55:07,584 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 614c85b0efc04e189067c3a9bd4ba3e8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1682567706344 2023-04-27 03:55:07,589 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e0314b11a1068d19a1052690701d1053#C#compaction#317 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:07,600 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/c81b2403f04341f08ba1c494a8a5b5e2 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/c81b2403f04341f08ba1c494a8a5b5e2 2023-04-27 03:55:07,601 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:07,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:07,604 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e0314b11a1068d19a1052690701d1053/C of e0314b11a1068d19a1052690701d1053 into c81b2403f04341f08ba1c494a8a5b5e2(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:07,604 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:07,604 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., storeName=e0314b11a1068d19a1052690701d1053/C, priority=13, startTime=1682567707542; duration=0sec 2023-04-27 03:55:07,604 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:07,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:07,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:07,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:07,606 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:07,606 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:07,606 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:07,606 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:07,606 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,606 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,608 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,608 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:07,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:07,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:07,608 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,608 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:07,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:07,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:07,610 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:07,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:07,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:07,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:07,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:07,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:07,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:07,610 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:07,610 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:07,610 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e0314b11a1068d19a1052690701d1053 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-27 03:55:07,610 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=A 2023-04-27 03:55:07,610 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:07,611 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=B 2023-04-27 03:55:07,611 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:07,611 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=C 2023-04-27 03:55:07,611 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:07,621 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304272cda16b072bd4e6bb056f2b93de7f792 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272cda16b072bd4e6bb056f2b93de7f792 2023-04-27 03:55:07,626 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=70, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/1c7c16c73f984b449bcbeac9a672465f 2023-04-27 03:55:07,649 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/d0177ad1e0694d43afc2d784fae38424 2023-04-27 03:55:07,666 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/5abe8125cada4ddb9a5959929aabc243 2023-04-27 03:55:07,672 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/1c7c16c73f984b449bcbeac9a672465f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/1c7c16c73f984b449bcbeac9a672465f 2023-04-27 03:55:07,675 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/1c7c16c73f984b449bcbeac9a672465f, entries=150, sequenceid=70, filesize=25.1 K 2023-04-27 03:55:07,676 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/d0177ad1e0694d43afc2d784fae38424 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/d0177ad1e0694d43afc2d784fae38424 2023-04-27 03:55:07,684 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/d0177ad1e0694d43afc2d784fae38424, entries=150, sequenceid=70, filesize=11.6 K 2023-04-27 03:55:07,685 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/5abe8125cada4ddb9a5959929aabc243 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/5abe8125cada4ddb9a5959929aabc243 2023-04-27 03:55:07,689 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/5abe8125cada4ddb9a5959929aabc243, entries=150, sequenceid=70, filesize=11.6 K 2023-04-27 03:55:07,690 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for e0314b11a1068d19a1052690701d1053 in 80ms, sequenceid=70, compaction requested=false 2023-04-27 03:55:07,690 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:07,690 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:07,690 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:07,690 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:07,690 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,690 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:07,690 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:07,692 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,692 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:07,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:07,692 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,692 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:07,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:07,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:07,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:07,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:07,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c62b5d6[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:07,694 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:07,694 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,697 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,697 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:07,697 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:07,698 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:07,698 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,698 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:07,699 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,699 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:07,699 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:07,699 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:07,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:07,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:07,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:07,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:07,702 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:07,702 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:07,702 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:07,702 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:07,702 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:07,703 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,703 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:07,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,703 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:07,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:07,704 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:07,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:07,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:07,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,705 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:07,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,708 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,708 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:07,708 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:07,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:07,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:07,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:07,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:07,708 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:07,708 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:07,708 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,708 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:07,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:07,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:07,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:07,708 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,708 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,709 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,709 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,709 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,808 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:07,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:07,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:07,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:07,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:07,812 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:07,812 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:07,812 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:07,812 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:07,812 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,812 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,815 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:07,815 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:07,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:07,815 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,815 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:07,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:07,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:07,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:07,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:07,816 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:07,816 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:07,816 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:07,817 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:07,817 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:07,817 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:07,817 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:07,817 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:07,817 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:07,817 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:07,817 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:07,817 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,817 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:07,817 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:07,819 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,819 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:07,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:07,819 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:07,819 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:07,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:07,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:07,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:07,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21b58893[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:07,821 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:07,821 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,822 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,822 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:07,822 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:07,822 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,822 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:07,822 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:07,824 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,824 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:07,824 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:07,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:07,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:07,824 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:07,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:07,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:07,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:07,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:07,826 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:07,827 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:07,827 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:07,827 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:07,827 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:07,829 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,829 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:07,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:07,829 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:07,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:07,830 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:07,831 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:07,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,833 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,833 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:07,833 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:07,833 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:07,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:07,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:07,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:07,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:07,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:07,833 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,834 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,834 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,834 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:07,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:07,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:07,834 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:07,834 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,834 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,834 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:07,876 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e0314b11a1068d19a1052690701d1053 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:55:07,877 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=A 2023-04-27 03:55:07,877 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:07,878 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=B 2023-04-27 03:55:07,878 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:07,878 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=C 2023-04-27 03:55:07,878 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:07,892 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304274743b2c3c8284211839dd2b5a6867809 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304274743b2c3c8284211839dd2b5a6867809 2023-04-27 03:55:07,908 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=82, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/d59e454f8a5543d1a799dda7e503fd88 2023-04-27 03:55:07,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:07,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59744 deadline: 1682567767909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:07,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567767910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:07,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567767918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:07,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567707905, max timestamp=1682567707919Num uniques 859; 2023-04-27 03:55:07,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567707905, max timestamp=1682567707919Num uniques 859; 2023-04-27 03:55:07,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:55:07,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567767918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:55:07,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567707905, max timestamp=1682567707919Num uniques 859; 2023-04-27 03:55:07,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:55:07,934 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:07,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:07,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:07,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:07,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:07,937 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:07,937 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:07,937 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:07,937 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:07,937 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,937 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,939 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:07,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:07,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:07,939 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,940 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,940 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:07,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:07,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:07,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:07,941 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:07,941 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:07,941 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:07,941 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:07,948 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:07,948 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:07,948 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:07,948 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:07,948 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:07,948 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:07,948 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:07,948 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:07,948 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,948 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:07,948 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:07,953 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,953 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:07,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:07,954 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,954 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:07,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:07,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:07,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:07,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:07,957 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:07,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17c45efd[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:07,957 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,958 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,958 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:07,958 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,958 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:07,958 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:07,958 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:07,959 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:07,959 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:07,960 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:07,960 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:07,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:07,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:07,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:07,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:07,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:07,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:07,962 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:07,962 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:07,962 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:07,962 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:07,962 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:07,963 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,964 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,964 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:07,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,964 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:07,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,965 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:07,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:07,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,970 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:07,970 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,970 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:07,970 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:07,970 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:07,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:07,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:07,970 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:07,970 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:07,970 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,970 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:07,970 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:07,970 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:07,970 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:07,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:07,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:07,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:07,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:07,996 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/d2cf3fa316da4a65a9d2437f8fd8bd7e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/d2cf3fa316da4a65a9d2437f8fd8bd7e 2023-04-27 03:55:08,000 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e0314b11a1068d19a1052690701d1053/A of e0314b11a1068d19a1052690701d1053 into d2cf3fa316da4a65a9d2437f8fd8bd7e(size=25.2 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:08,000 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:08,000 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., storeName=e0314b11a1068d19a1052690701d1053/A, priority=13, startTime=1682567707542; duration=0sec 2023-04-27 03:55:08,000 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:08,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:08,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567768022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:08,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567768024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:08,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567768028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:08,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59744 deadline: 1682567768029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:08,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567768029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,070 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:08,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:08,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:08,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:08,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:08,074 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:08,074 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:08,074 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:08,074 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:08,074 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,074 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,076 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:08,076 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:08,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:08,076 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,076 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:08,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:08,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:08,077 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:08,077 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:08,077 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:08,077 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:08,078 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:08,078 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:08,078 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:08,078 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:08,078 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,078 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:08,078 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:08,079 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,079 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:08,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:08,080 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,080 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:08,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:08,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:08,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:08,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:08,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d0c746d[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:08,081 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:08,081 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,082 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,082 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:08,082 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,082 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:08,082 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:08,082 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:08,085 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,085 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:08,085 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:08,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:08,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:08,085 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:08,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:08,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:08,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:08,087 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:08,087 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:08,087 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:08,087 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:08,087 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:08,088 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,088 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:08,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,089 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:08,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:08,089 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:08,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,090 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:08,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,093 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:08,093 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,093 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:08,093 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:08,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:08,093 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:08,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:08,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:08,093 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,093 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,093 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,093 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,093 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,094 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:08,094 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:08,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:08,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:08,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:08,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567768137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:08,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59744 deadline: 1682567768140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:08,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567768140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,194 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:08,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:08,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:08,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:08,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:08,197 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:08,197 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:08,197 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:08,197 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:08,198 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,198 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,199 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:08,199 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:08,199 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,199 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:08,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:08,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:08,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:08,200 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:08,200 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:08,200 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:08,201 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:08,201 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:08,201 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:08,201 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:08,201 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:08,201 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:08,201 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:08,201 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:08,201 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,201 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:08,201 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:08,204 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,204 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:08,204 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,204 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:08,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:08,205 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:08,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5439b81f[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:08,205 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,207 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,207 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:08,207 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:08,207 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,207 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:08,207 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:08,208 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,208 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:08,208 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:08,208 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:08,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:08,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:08,211 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:08,211 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:08,211 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:08,211 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:08,211 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:08,212 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,212 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:08,212 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:08,213 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:08,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,214 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:08,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,220 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,220 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,220 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,220 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,220 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,220 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,220 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,220 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:08,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:08,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:08,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:08,220 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:08,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:08,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:08,220 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:08,220 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:08,221 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:08,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:08,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:08,235 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:08,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567768230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:08,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567768233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,318 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/8e8d071a65f846adabbbd43e761ea8f7 2023-04-27 03:55:08,321 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:08,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:08,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:08,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:08,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:08,324 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:08,324 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:08,324 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:08,324 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:08,324 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,324 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,325 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:08,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:08,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:08,325 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,326 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,326 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:08,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:08,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:08,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:08,327 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:08,327 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:08,327 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:08,327 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:08,327 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:08,328 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:08,328 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:08,328 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:08,328 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:08,328 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:08,328 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:08,328 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:08,328 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,328 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:08,328 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:08,329 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,329 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:08,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:08,330 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,330 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:08,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:08,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59c5431b[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:08,331 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:08,331 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,337 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,337 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,338 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:08,337 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:08,338 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:08,338 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:08,339 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,339 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:08,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:08,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:08,339 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:08,339 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:08,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:08,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:08,341 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:08,341 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:08,341 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:08,341 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:08,341 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:08,343 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,343 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,343 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:08,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:08,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:08,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:08,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:08,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:08,343 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,345 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,347 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,347 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,347 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,347 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,347 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:08,347 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:08,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:08,347 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,347 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:08,347 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,347 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:08,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:08,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:08,347 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:08,347 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:08,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:08,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:08,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:08,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567768342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:08,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59744 deadline: 1682567768343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:08,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567768343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,447 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:08,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:08,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:08,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:08,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:08,450 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:08,450 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:08,450 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:08,450 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:08,451 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,451 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,453 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:08,453 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:08,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:08,454 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,454 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:08,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:08,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:08,455 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:08,455 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:08,455 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:08,455 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:08,455 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:08,455 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:08,455 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:08,455 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:08,456 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:08,456 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:08,456 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:08,456 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,456 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:08,456 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:08,457 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,457 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:08,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:08,457 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,457 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:08,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:08,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:08,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cc3dd68[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:08,459 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:08,459 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,460 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,460 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:08,460 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:08,460 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:08,460 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,460 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:08,461 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,461 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:08,461 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:08,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:08,461 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:08,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:08,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:08,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:08,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:08,465 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:08,465 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:08,465 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:08,465 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:08,465 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:08,468 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,468 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:08,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,468 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:08,468 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:08,469 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,471 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,471 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,471 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,472 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:08,472 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:08,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:08,472 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,472 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:08,472 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:08,472 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:08,472 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,472 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:08,472 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:08,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:08,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:08,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:08,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567768536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:08,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567768536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,572 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:08,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:08,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:08,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:08,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:08,575 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:08,575 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:08,575 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:08,575 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:08,576 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,576 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,577 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:08,577 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:08,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:08,577 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,577 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:08,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:08,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:08,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:08,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:08,578 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:08,578 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:08,579 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:08,579 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:08,579 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:08,579 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:08,579 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:08,579 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:08,579 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:08,579 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:08,579 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:08,579 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,579 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:08,579 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:08,581 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,581 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:08,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:08,581 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,581 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:08,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:08,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:08,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:08,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:08,582 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:08,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22227381[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:08,583 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,585 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,585 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:08,585 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:08,585 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,585 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:08,585 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:08,586 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,586 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:08,586 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:08,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:08,586 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:08,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:08,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:08,589 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:08,589 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:08,589 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:08,589 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:08,589 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:08,590 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,590 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:08,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:08,590 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:08,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:08,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:08,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:08,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,591 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,592 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,594 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,594 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:08,594 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:08,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:08,594 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:08,595 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:08,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:08,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:08,594 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,595 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,595 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,595 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,595 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,595 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:08,595 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:08,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:08,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:08,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567768655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:08,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59744 deadline: 1682567768655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:08,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567768656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,695 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:08,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:08,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:08,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:08,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:08,698 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:08,698 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:08,698 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:08,698 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:08,699 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,699 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,702 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,702 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:08,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:08,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:08,702 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,702 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:08,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:08,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:08,703 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:08,703 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:08,703 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:08,703 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:08,703 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:08,703 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:08,703 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:08,703 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:08,704 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:08,704 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:08,704 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:08,704 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:08,704 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:08,704 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,704 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:08,704 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:08,706 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,706 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:08,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:08,706 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,706 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:08,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:08,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:08,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bc2de9a[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:08,707 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:08,708 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,709 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,709 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:08,709 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:08,709 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:08,709 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,709 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:08,710 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,710 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:08,710 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:08,710 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:08,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:08,712 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:08,712 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:08,712 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:08,713 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:08,713 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:08,718 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,718 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:08,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,718 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:08,718 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:08,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,720 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,722 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:08,722 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:08,722 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:08,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:08,722 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:08,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:08,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:08,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:08,722 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,722 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:08,722 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:08,722 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,722 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,722 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:08,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:08,722 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:08,722 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,728 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/f06f13eff0354fd4ae6d4b5d2fb4a4b9 2023-04-27 03:55:08,732 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/d59e454f8a5543d1a799dda7e503fd88 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/d59e454f8a5543d1a799dda7e503fd88 2023-04-27 03:55:08,737 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/d59e454f8a5543d1a799dda7e503fd88, entries=400, sequenceid=82, filesize=59.0 K 2023-04-27 03:55:08,737 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/8e8d071a65f846adabbbd43e761ea8f7 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/8e8d071a65f846adabbbd43e761ea8f7 2023-04-27 03:55:08,753 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/8e8d071a65f846adabbbd43e761ea8f7, entries=150, sequenceid=82, filesize=11.6 K 2023-04-27 03:55:08,754 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/f06f13eff0354fd4ae6d4b5d2fb4a4b9 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/f06f13eff0354fd4ae6d4b5d2fb4a4b9 2023-04-27 03:55:08,758 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/f06f13eff0354fd4ae6d4b5d2fb4a4b9, entries=150, sequenceid=82, filesize=11.6 K 2023-04-27 03:55:08,758 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e0314b11a1068d19a1052690701d1053 in 882ms, sequenceid=82, compaction requested=true 2023-04-27 03:55:08,758 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:08,758 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:08,759 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:08,759 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:08,759 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:08,759 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:08,760 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 112049 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:08,760 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): e0314b11a1068d19a1052690701d1053/A is initiating minor compaction (all files) 2023-04-27 03:55:08,760 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:08,760 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e0314b11a1068d19a1052690701d1053/A in TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:08,760 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e0314b11a1068d19a1052690701d1053/B is initiating minor compaction (all files) 2023-04-27 03:55:08,760 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/d2cf3fa316da4a65a9d2437f8fd8bd7e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/1c7c16c73f984b449bcbeac9a672465f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/d59e454f8a5543d1a799dda7e503fd88] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp, totalSize=109.4 K 2023-04-27 03:55:08,760 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e0314b11a1068d19a1052690701d1053/B in TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:08,760 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/f8aa684a3260406eb20279223e03caea, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/d0177ad1e0694d43afc2d784fae38424, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/8e8d071a65f846adabbbd43e761ea8f7] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp, totalSize=34.9 K 2023-04-27 03:55:08,761 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting d2cf3fa316da4a65a9d2437f8fd8bd7e, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1682567706344 2023-04-27 03:55:08,761 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting f8aa684a3260406eb20279223e03caea, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1682567706344 2023-04-27 03:55:08,761 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 1c7c16c73f984b449bcbeac9a672465f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682567706686 2023-04-27 03:55:08,761 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting d0177ad1e0694d43afc2d784fae38424, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682567706686 2023-04-27 03:55:08,761 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting d59e454f8a5543d1a799dda7e503fd88, keycount=400, bloomtype=ROW, size=59.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682567707864 2023-04-27 03:55:08,761 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 8e8d071a65f846adabbbd43e761ea8f7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682567707866 2023-04-27 03:55:08,768 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e0314b11a1068d19a1052690701d1053#B#compaction#325 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:08,773 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): e0314b11a1068d19a1052690701d1053#A#compaction#324 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:08,822 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:08,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:08,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:08,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:08,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:08,825 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:08,825 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:08,825 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:08,825 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:08,826 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,826 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,827 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:08,827 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:08,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:08,827 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:08,827 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:08,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:08,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:08,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:08,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:08,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:08,828 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:08,828 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:08,828 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:08,829 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:08,829 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:08,829 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:08,829 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:08,829 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e0314b11a1068d19a1052690701d1053 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-27 03:55:08,829 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=A 2023-04-27 03:55:08,829 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:08,829 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=B 2023-04-27 03:55:08,829 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:08,829 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=C 2023-04-27 03:55:08,829 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:09,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:09,050 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:09,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:09,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567769070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:09,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567769071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:09,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567769159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:09,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59744 deadline: 1682567769160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:09,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567769162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:09,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567769178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:09,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567769184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,189 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/4d9d7ac79ab141578a1d68aef018ec92 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/4d9d7ac79ab141578a1d68aef018ec92 2023-04-27 03:55:09,194 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e0314b11a1068d19a1052690701d1053/B of e0314b11a1068d19a1052690701d1053 into 4d9d7ac79ab141578a1d68aef018ec92(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:09,194 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:09,194 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., storeName=e0314b11a1068d19a1052690701d1053/B, priority=13, startTime=1682567708759; duration=0sec 2023-04-27 03:55:09,194 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:09,194 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:09,195 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:09,195 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e0314b11a1068d19a1052690701d1053/C is initiating minor compaction (all files) 2023-04-27 03:55:09,195 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e0314b11a1068d19a1052690701d1053/C in TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:09,195 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/c81b2403f04341f08ba1c494a8a5b5e2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/5abe8125cada4ddb9a5959929aabc243, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/f06f13eff0354fd4ae6d4b5d2fb4a4b9] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp, totalSize=34.9 K 2023-04-27 03:55:09,196 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting c81b2403f04341f08ba1c494a8a5b5e2, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1682567706344 2023-04-27 03:55:09,196 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 5abe8125cada4ddb9a5959929aabc243, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682567706686 2023-04-27 03:55:09,197 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting f06f13eff0354fd4ae6d4b5d2fb4a4b9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682567707866 2023-04-27 03:55:09,206 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e0314b11a1068d19a1052690701d1053#C#compaction#327 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:09,213 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/3219a1ca6b894c2cb3372d07413a01f9 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/3219a1ca6b894c2cb3372d07413a01f9 2023-04-27 03:55:09,217 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e0314b11a1068d19a1052690701d1053/C of e0314b11a1068d19a1052690701d1053 into 3219a1ca6b894c2cb3372d07413a01f9(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:09,217 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:09,217 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., storeName=e0314b11a1068d19a1052690701d1053/C, priority=13, startTime=1682567708759; duration=0sec 2023-04-27 03:55:09,217 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:09,246 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427f8055d443bf5469097ea7239dbeff605 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427f8055d443bf5469097ea7239dbeff605 2023-04-27 03:55:09,258 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=118, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/f24e1932c8fe4444857a58e3d33ef6da 2023-04-27 03:55:09,270 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/76609995604646f299e5f56b46cf75ad 2023-04-27 03:55:09,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:09,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567769382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:09,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567769391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,598 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/36647eebafe2494fa84d245032b9e151 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/36647eebafe2494fa84d245032b9e151 2023-04-27 03:55:09,602 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e0314b11a1068d19a1052690701d1053/A of e0314b11a1068d19a1052690701d1053 into 36647eebafe2494fa84d245032b9e151(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:09,602 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:09,602 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., storeName=e0314b11a1068d19a1052690701d1053/A, priority=13, startTime=1682567708758; duration=0sec 2023-04-27 03:55:09,602 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:09,681 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/a74fb38ab1e340f5a9ed80b2da779585 2023-04-27 03:55:09,684 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/f24e1932c8fe4444857a58e3d33ef6da as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/f24e1932c8fe4444857a58e3d33ef6da 2023-04-27 03:55:09,688 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/f24e1932c8fe4444857a58e3d33ef6da, entries=150, sequenceid=118, filesize=25.1 K 2023-04-27 03:55:09,688 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/76609995604646f299e5f56b46cf75ad as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/76609995604646f299e5f56b46cf75ad 2023-04-27 03:55:09,692 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/76609995604646f299e5f56b46cf75ad, entries=150, sequenceid=118, filesize=11.6 K 2023-04-27 03:55:09,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:09,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567769687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,692 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/a74fb38ab1e340f5a9ed80b2da779585 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/a74fb38ab1e340f5a9ed80b2da779585 2023-04-27 03:55:09,695 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/a74fb38ab1e340f5a9ed80b2da779585, entries=150, sequenceid=118, filesize=11.6 K 2023-04-27 03:55:09,696 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for e0314b11a1068d19a1052690701d1053 in 867ms, sequenceid=118, compaction requested=false 2023-04-27 03:55:09,696 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:09,696 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:09,696 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:09,696 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:09,696 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,696 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:09,696 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:09,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:09,698 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e0314b11a1068d19a1052690701d1053 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-27 03:55:09,698 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=A 2023-04-27 03:55:09,699 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:09,699 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=B 2023-04-27 03:55:09,699 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:09,699 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=C 2023-04-27 03:55:09,699 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:09,699 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:09,699 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:09,699 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:09,699 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:09,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:09,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:09,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:09,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:09,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:09,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:09,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48ec8f62[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:09,702 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:09,702 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:09,705 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,705 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:09,707 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:09,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:09,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:09,707 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:09,708 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:09,708 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:09,709 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:09,709 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:09,709 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:09,709 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:09,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:09,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:09,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:09,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:09,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:09,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:09,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:09,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:09,711 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:09,711 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:09,711 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:09,711 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:09,711 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:09,713 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427a45a7f4ad0b1466089a3508930741523 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427a45a7f4ad0b1466089a3508930741523 2023-04-27 03:55:09,713 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,713 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:09,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,713 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:09,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:09,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:09,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:09,714 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:09,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:09,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:09,716 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:09,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:09,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,718 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:09,718 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:09,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:09,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:09,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:09,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:09,719 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,719 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:09,719 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:09,719 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:09,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:09,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:09,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:09,719 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,720 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:09,720 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:09,720 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:09,720 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:09,720 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:09,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567769775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,820 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:09,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:09,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:09,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:09,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:09,823 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:09,823 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:09,823 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:09,823 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:09,823 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,823 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:09,825 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:09,825 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:09,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:09,826 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,826 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:09,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:09,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:09,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:09,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:09,827 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:09,827 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:09,827 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:09,827 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:09,827 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:09,827 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:09,827 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:09,827 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:09,827 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:09,827 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:09,827 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:09,827 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,827 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:09,827 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:09,829 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,829 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:09,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:09,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:09,829 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:09,829 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:09,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:09,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:09,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:09,830 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:09,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a36338a[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:09,830 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:09,831 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:09,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:09,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:09,832 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:09,832 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:09,832 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,832 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:09,832 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:09,833 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,833 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:09,833 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:09,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:09,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:09,833 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:09,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:09,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:09,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:09,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:09,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:09,835 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:09,836 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:09,836 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:09,836 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:09,836 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:09,838 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:09,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:09,838 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,838 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:09,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:09,839 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:09,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:09,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:09,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:09,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:09,840 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:09,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:09,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,842 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:09,842 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:09,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:09,842 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:09,842 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:09,842 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:09,842 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:09,842 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:09,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:09,842 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:09,842 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:09,842 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,842 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:09,843 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:09,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:09,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:09,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:09,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:09,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567769881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,943 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:09,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:09,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:09,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:09,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:09,946 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:09,946 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:09,946 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:09,946 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:09,946 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,946 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:09,947 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,947 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:09,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:09,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:09,948 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,948 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:09,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:09,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:09,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:09,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:09,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:09,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:09,949 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:09,949 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:09,949 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:09,950 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:09,950 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:09,950 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:09,950 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:09,950 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:09,950 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:09,950 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:09,951 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:09,951 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,951 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:09,951 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:09,952 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,952 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:09,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:09,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:09,953 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:09,953 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:09,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:09,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:09,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:09,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:09,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:09,954 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:09,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b39e136[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:09,954 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:09,957 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:09,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:09,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:09,957 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:09,957 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:09,957 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:09,957 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,957 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:09,958 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:09,958 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:09,958 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,958 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:09,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:09,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:09,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:09,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:09,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:09,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:09,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:09,961 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:09,961 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:09,962 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:09,962 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:09,962 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:09,963 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,963 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,963 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:09,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:09,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:09,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:09,964 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:09,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:09,965 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:09,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:09,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,972 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,972 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:09,972 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:09,972 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:09,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:09,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:09,972 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:09,972 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:09,972 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:09,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:09,972 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:09,972 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:09,972 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:09,972 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:09,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:09,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:09,972 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:09,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:09,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:10,072 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:10,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:10,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:10,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:10,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:10,076 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:10,076 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:10,076 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:10,076 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:10,076 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,076 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:10,077 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:10,077 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:10,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:10,078 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,078 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:10,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:10,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:10,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:10,079 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:10,079 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:10,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:10,079 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:10,079 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:10,079 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:10,079 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:10,079 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:10,079 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:10,079 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:10,079 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:10,079 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:10,080 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:10,080 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:10,080 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,080 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:10,080 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:10,081 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,081 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:10,081 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,081 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:10,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:10,083 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:10,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26a2d8c5[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:10,083 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,084 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,084 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:10,084 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:10,084 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:10,084 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,084 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:10,088 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,088 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:10,088 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:10,088 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:10,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:10,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:10,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:10,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567770087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:10,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:10,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:10,090 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:10,090 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:10,090 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:10,090 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:10,090 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:10,092 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,092 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:10,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:10,092 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:10,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:10,092 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:10,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:10,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:10,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:10,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:10,094 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:10,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:10,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,095 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,095 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:10,095 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:10,096 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:10,096 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:10,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:10,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:10,096 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,096 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,096 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,096 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:10,096 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:10,096 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:10,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:10,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:10,096 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:10,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:10,118 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=133, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/ffdff68fae094f81a8b280055b3bd294 2023-04-27 03:55:10,128 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/e2ab8078edd64bdf89d9f51d9ff662d0 2023-04-27 03:55:10,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:10,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567770165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:10,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567770170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:10,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59744 deadline: 1682567770171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,196 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:10,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:10,198 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:10,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567770194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:10,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:10,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:10,199 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:10,199 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:10,199 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:10,199 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:10,199 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,199 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:10,201 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:10,201 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:10,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:10,201 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:10,201 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:10,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:10,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:10,202 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:10,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:10,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:10,202 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:10,202 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:10,202 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:10,202 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:10,202 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:10,202 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:10,202 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:10,202 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:10,202 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:10,202 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:10,202 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:10,202 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,202 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:10,202 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:10,205 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,205 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:10,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:10,205 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,205 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:10,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:10,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:10,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:10,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:10,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:10,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a19c4ec[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:10,206 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:10,206 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,207 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,208 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:10,208 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:10,208 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:10,208 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,208 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:10,209 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,209 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:10,209 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:10,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:10,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:10,209 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:10,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:10,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:10,211 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:10,211 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:10,211 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:10,211 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:10,211 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:10,213 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,213 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:10,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,213 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:10,213 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:10,214 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,217 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,217 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:10,217 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:10,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:10,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:10,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:10,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:10,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:10,217 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:10,217 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:10,218 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:10,218 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:10,217 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:10,218 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:10,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:10,218 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,218 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,218 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,318 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:10,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:10,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:10,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:10,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:10,321 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:10,321 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:10,321 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:10,321 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:10,321 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,321 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:10,323 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:10,323 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:10,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:10,323 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,323 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:10,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:10,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:10,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:10,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:10,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:10,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:10,324 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:10,324 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:10,324 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:10,325 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:10,325 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:10,325 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:10,325 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:10,325 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:10,325 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:10,325 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:10,325 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:10,325 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,325 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:10,325 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:10,326 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,326 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:10,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:10,327 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,327 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:10,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:10,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:10,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:10,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:10,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:10,328 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:10,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b81c46[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:10,329 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,329 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,330 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:10,330 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,330 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:10,330 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:10,330 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:10,332 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:10,333 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,333 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:10,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:10,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:10,333 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:10,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:10,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:10,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:10,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:10,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:10,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:10,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:10,335 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:10,335 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:10,335 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:10,335 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:10,335 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:10,336 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,336 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:10,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:10,336 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:10,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:10,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:10,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:10,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,337 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:10,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:10,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:10,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,338 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:10,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:10,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,343 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:10,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:10,343 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:10,343 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:10,343 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:10,343 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:10,343 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:10,343 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,343 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,343 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,343 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,343 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:10,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:10,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:10,343 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:10,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:10,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:10,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:10,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567770392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,443 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:10,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:10,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:10,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:10,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:10,447 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:10,447 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:10,447 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:10,447 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:10,447 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,447 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:10,449 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:10,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:10,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:10,449 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:10,450 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,450 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:10,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:10,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:10,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:10,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:10,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:10,451 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:10,451 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:10,451 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:10,451 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:10,451 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:10,451 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:10,451 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:10,451 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:10,451 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:10,451 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:10,451 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:10,451 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,451 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:10,451 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:10,455 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:10,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:10,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:10,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:10,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:10,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:10,456 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:10,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@454c6d16[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:10,456 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,457 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,458 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:10,458 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:10,458 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:10,458 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,458 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:10,459 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,459 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:10,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:10,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:10,459 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:10,459 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:10,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:10,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:10,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:10,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:10,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:10,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:10,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:10,461 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:10,461 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:10,461 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:10,461 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:10,461 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:10,466 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,466 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:10,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:10,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,466 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:10,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:10,467 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:10,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:10,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:10,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:10,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:10,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,468 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,470 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,470 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:10,470 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:10,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:10,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:10,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:10,470 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:10,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:10,470 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:10,470 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:10,470 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,471 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,471 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:10,471 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:10,471 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:10,471 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:10,537 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/cd2f6ce233e74c50804b2fd00e95cb75 2023-04-27 03:55:10,540 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/ffdff68fae094f81a8b280055b3bd294 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/ffdff68fae094f81a8b280055b3bd294 2023-04-27 03:55:10,544 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/ffdff68fae094f81a8b280055b3bd294, entries=200, sequenceid=133, filesize=32.0 K 2023-04-27 03:55:10,544 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/e2ab8078edd64bdf89d9f51d9ff662d0 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/e2ab8078edd64bdf89d9f51d9ff662d0 2023-04-27 03:55:10,547 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/e2ab8078edd64bdf89d9f51d9ff662d0, entries=150, sequenceid=133, filesize=11.7 K 2023-04-27 03:55:10,548 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/cd2f6ce233e74c50804b2fd00e95cb75 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/cd2f6ce233e74c50804b2fd00e95cb75 2023-04-27 03:55:10,551 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/cd2f6ce233e74c50804b2fd00e95cb75, entries=150, sequenceid=133, filesize=11.7 K 2023-04-27 03:55:10,552 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for e0314b11a1068d19a1052690701d1053 in 854ms, sequenceid=133, compaction requested=true 2023-04-27 03:55:10,552 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:10,552 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:10,552 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:10,552 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:10,552 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:10,553 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:10,554 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84434 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:10,554 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35914 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:10,554 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): e0314b11a1068d19a1052690701d1053/A is initiating minor compaction (all files) 2023-04-27 03:55:10,554 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e0314b11a1068d19a1052690701d1053/B is initiating minor compaction (all files) 2023-04-27 03:55:10,554 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e0314b11a1068d19a1052690701d1053/A in TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:10,554 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e0314b11a1068d19a1052690701d1053/B in TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:10,554 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/36647eebafe2494fa84d245032b9e151, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/f24e1932c8fe4444857a58e3d33ef6da, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/ffdff68fae094f81a8b280055b3bd294] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp, totalSize=82.5 K 2023-04-27 03:55:10,554 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/4d9d7ac79ab141578a1d68aef018ec92, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/76609995604646f299e5f56b46cf75ad, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/e2ab8078edd64bdf89d9f51d9ff662d0] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp, totalSize=35.1 K 2023-04-27 03:55:10,554 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 36647eebafe2494fa84d245032b9e151, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682567707866 2023-04-27 03:55:10,554 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 4d9d7ac79ab141578a1d68aef018ec92, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682567707866 2023-04-27 03:55:10,554 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting f24e1932c8fe4444857a58e3d33ef6da, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682567708010 2023-04-27 03:55:10,555 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 76609995604646f299e5f56b46cf75ad, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682567708010 2023-04-27 03:55:10,555 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting ffdff68fae094f81a8b280055b3bd294, keycount=200, bloomtype=ROW, size=32.0 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1682567709049 2023-04-27 03:55:10,555 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting e2ab8078edd64bdf89d9f51d9ff662d0, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1682567709049 2023-04-27 03:55:10,565 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e0314b11a1068d19a1052690701d1053#B#compaction#334 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:10,571 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:10,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:10,573 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): e0314b11a1068d19a1052690701d1053#A#compaction#333 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:10,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:10,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:10,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:10,575 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:10,575 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:10,575 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:10,575 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:10,576 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,576 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:10,578 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:10,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:10,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:10,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:10,578 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:10,579 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,579 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:10,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:10,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:10,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:10,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:10,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:10,580 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:10,580 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:10,580 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:10,581 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:10,581 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:10,581 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:10,581 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:10,582 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e0314b11a1068d19a1052690701d1053 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-27 03:55:10,582 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=A 2023-04-27 03:55:10,582 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:10,582 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=B 2023-04-27 03:55:10,582 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:10,582 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=C 2023-04-27 03:55:10,582 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:10,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:10,901 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:10,954 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:10,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567770949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:10,979 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/68301c30cbe1434597c3b2e4345eba3f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/68301c30cbe1434597c3b2e4345eba3f 2023-04-27 03:55:10,983 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e0314b11a1068d19a1052690701d1053/B of e0314b11a1068d19a1052690701d1053 into 68301c30cbe1434597c3b2e4345eba3f(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:10,983 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:10,983 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., storeName=e0314b11a1068d19a1052690701d1053/B, priority=13, startTime=1682567710552; duration=0sec 2023-04-27 03:55:10,983 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:10,983 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:10,984 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35914 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:10,984 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e0314b11a1068d19a1052690701d1053/C is initiating minor compaction (all files) 2023-04-27 03:55:10,984 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e0314b11a1068d19a1052690701d1053/C in TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:10,984 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/3219a1ca6b894c2cb3372d07413a01f9, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/a74fb38ab1e340f5a9ed80b2da779585, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/cd2f6ce233e74c50804b2fd00e95cb75] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp, totalSize=35.1 K 2023-04-27 03:55:10,985 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 3219a1ca6b894c2cb3372d07413a01f9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682567707866 2023-04-27 03:55:10,985 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting a74fb38ab1e340f5a9ed80b2da779585, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682567708010 2023-04-27 03:55:10,985 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting cd2f6ce233e74c50804b2fd00e95cb75, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1682567709049 2023-04-27 03:55:10,992 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e0314b11a1068d19a1052690701d1053#C#compaction#336 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:11,001 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304274bc82964c1594c56837a6ca0ec73cf72 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304274bc82964c1594c56837a6ca0ec73cf72 2023-04-27 03:55:11,017 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=153, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/b7a7f1d9c640483d8e059408d51f2b52 2023-04-27 03:55:11,019 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/35f8386ff0fb48f3b20c27cd2aec6d44 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/35f8386ff0fb48f3b20c27cd2aec6d44 2023-04-27 03:55:11,021 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/7504b76793e44df8af079aec034474b5 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/7504b76793e44df8af079aec034474b5 2023-04-27 03:55:11,028 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e0314b11a1068d19a1052690701d1053/C of e0314b11a1068d19a1052690701d1053 into 7504b76793e44df8af079aec034474b5(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:11,028 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:11,028 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., storeName=e0314b11a1068d19a1052690701d1053/C, priority=13, startTime=1682567710553; duration=0sec 2023-04-27 03:55:11,028 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:11,030 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e0314b11a1068d19a1052690701d1053/A of e0314b11a1068d19a1052690701d1053 into 35f8386ff0fb48f3b20c27cd2aec6d44(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:11,030 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:11,030 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., storeName=e0314b11a1068d19a1052690701d1053/A, priority=13, startTime=1682567710552; duration=0sec 2023-04-27 03:55:11,030 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:11,035 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/98ec8be65f11438d896ecca506d0a6ac 2023-04-27 03:55:11,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:11,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567771055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:11,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567771199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:11,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567771261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,451 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/9726a89c9b7b49bd9e016efaf6d27da5 2023-04-27 03:55:11,454 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/b7a7f1d9c640483d8e059408d51f2b52 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/b7a7f1d9c640483d8e059408d51f2b52 2023-04-27 03:55:11,458 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/b7a7f1d9c640483d8e059408d51f2b52, entries=150, sequenceid=153, filesize=25.3 K 2023-04-27 03:55:11,458 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/98ec8be65f11438d896ecca506d0a6ac as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/98ec8be65f11438d896ecca506d0a6ac 2023-04-27 03:55:11,462 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/98ec8be65f11438d896ecca506d0a6ac, entries=150, sequenceid=153, filesize=11.8 K 2023-04-27 03:55:11,463 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/9726a89c9b7b49bd9e016efaf6d27da5 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/9726a89c9b7b49bd9e016efaf6d27da5 2023-04-27 03:55:11,466 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/9726a89c9b7b49bd9e016efaf6d27da5, entries=150, sequenceid=153, filesize=11.8 K 2023-04-27 03:55:11,466 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for e0314b11a1068d19a1052690701d1053 in 884ms, sequenceid=153, compaction requested=false 2023-04-27 03:55:11,466 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:11,466 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:11,466 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:11,467 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:11,467 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,467 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:11,467 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:11,468 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,468 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:11,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:11,468 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,469 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:11,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:11,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:11,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:11,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:11,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7598f18a[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:11,470 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:11,470 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,475 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,475 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:11,475 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:11,475 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:11,475 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,475 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:11,476 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:11,476 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,476 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:11,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:11,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:11,476 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:11,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:11,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:11,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:11,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:11,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:11,479 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:11,479 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:11,479 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:11,479 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:11,479 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:11,480 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,480 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:11,480 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:11,481 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:11,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:11,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:11,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,482 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:11,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:11,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:11,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:11,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:11,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:11,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:11,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:11,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:11,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:11,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,485 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,485 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,485 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:11,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:11,485 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:11,485 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:11,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:11,579 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e0314b11a1068d19a1052690701d1053 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-27 03:55:11,579 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=A 2023-04-27 03:55:11,579 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:11,579 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=B 2023-04-27 03:55:11,579 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:11,579 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=C 2023-04-27 03:55:11,580 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:11,585 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:11,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:11,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:11,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:11,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:11,590 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:11,590 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:11,590 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:11,590 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:11,591 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,591 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:11,604 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,604 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:11,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:11,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:11,604 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,605 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:11,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:11,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:11,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:11,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:11,605 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:11,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:11,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:11,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:11,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:11,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:11,606 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:11,606 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:11,606 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:11,606 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:11,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:11,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:11,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:11,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:11,608 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,608 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:11,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:11,608 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,608 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:11,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:11,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:11,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:11,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:11,610 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:11,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31e27571[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:11,610 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,611 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,611 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:11,611 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,611 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:11,611 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:11,611 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:11,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,612 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:11,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:11,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:11,612 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:11,612 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:11,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:11,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:11,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:11,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:11,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:11,615 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:11,615 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:11,615 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:11,615 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:11,615 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:11,617 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,617 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:11,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:11,617 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:11,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:11,618 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:11,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:11,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:11,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,619 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:11,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,621 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:11,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:11,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:11,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:11,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:11,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:11,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:11,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:11,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:11,622 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:11,622 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:11,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:11,622 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:11,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:11,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,722 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:11,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:11,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:11,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:11,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:11,726 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:11,726 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:11,726 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:11,726 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:11,726 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,726 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:11,727 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:11,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:11,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:11,728 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,728 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:11,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:11,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:11,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:11,728 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:11,729 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:11,729 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:11,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:11,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:11,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:11,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:11,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:11,729 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:11,729 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:11,729 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:11,729 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:11,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:11,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:11,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:11,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:11,734 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,734 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:11,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:11,734 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,734 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:11,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:11,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:11,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:11,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:11,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:11,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70f91527[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:11,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,738 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:11,738 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:11,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:11,738 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:11,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:11,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:11,739 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:11,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:11,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:11,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:11,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:11,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:11,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:11,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:11,742 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:11,742 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:11,742 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:11,742 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:11,742 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:11,743 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,743 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:11,743 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:11,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:11,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:11,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:11,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:11,744 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:11,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,746 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:11,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,748 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:11,748 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:11,748 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:11,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:11,748 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:11,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:11,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:11,748 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:11,748 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:11,748 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,748 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,748 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,748 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:11,749 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:11,748 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:11,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:11,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:11,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567771767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,849 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:11,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:11,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:11,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:11,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:11,852 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:11,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:11,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:11,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:11,853 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,853 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:11,855 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:11,855 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:11,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:11,855 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,855 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:11,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:11,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:11,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:11,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:11,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:11,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:11,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:11,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:11,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:11,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:11,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:11,856 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:11,857 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:11,857 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:11,857 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:11,857 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:11,857 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:11,857 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,857 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:11,857 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:11,858 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,858 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:11,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:11,858 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,859 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:11,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:11,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:11,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:11,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:11,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56fbf2f3[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:11,860 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:11,860 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,861 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,861 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:11,861 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:11,861 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:11,861 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,861 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:11,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:11,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:11,862 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:11,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:11,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:11,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:11,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:11,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:11,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:11,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:11,865 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:11,865 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:11,865 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:11,865 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:11,865 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:11,871 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,871 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,871 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:11,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:11,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:11,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:11,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:11,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,872 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:11,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:11,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,873 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:11,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,876 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,876 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:11,876 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:11,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:11,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:11,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:11,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:11,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:11,876 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:11,876 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:11,876 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:11,876 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:11,876 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:11,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:11,876 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:11,876 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,877 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,877 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:11,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567771874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,977 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:11,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:11,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:11,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:11,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:11,980 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:11,980 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:11,980 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:11,980 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:11,980 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,980 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:11,981 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,981 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:11,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:11,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:11,982 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,982 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:11,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:11,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:11,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:11,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:11,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:11,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:11,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:11,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:11,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:11,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:11,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:11,983 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:11,983 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:11,983 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:11,983 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:11,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:11,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:11,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:11,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:11,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,985 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:11,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:11,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:11,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:11,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:11,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:11,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:11,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64dd5d07[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:11,986 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:11,987 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,989 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:11,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:11,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:11,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:11,989 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,989 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:11,991 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,991 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:11,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:11,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:11,991 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:11,991 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:11,991 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427f65e6e43f54a48809580d8c03e70809f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427f65e6e43f54a48809580d8c03e70809f 2023-04-27 03:55:11,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:11,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:11,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:11,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:11,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:11,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:11,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:11,998 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:11,998 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:11,998 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:11,998 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:11,998 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:11,999 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,999 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,999 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:11,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:11,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:11,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:12,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:12,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:12,002 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:12,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:12,003 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,005 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,005 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:12,005 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:12,005 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,005 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,005 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:12,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:12,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:12,005 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,005 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,006 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:12,006 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:12,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:12,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:12,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:12,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:12,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:12,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:12,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:12,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567772079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,106 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:12,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:12,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:12,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:12,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:12,109 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:12,109 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:12,109 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:12,109 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:12,109 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,109 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:12,112 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:12,112 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:12,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:12,112 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,112 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:12,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:12,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:12,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:12,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:12,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:12,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:12,113 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:12,113 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:12,113 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:12,113 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:12,113 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:12,113 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:12,113 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:12,113 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:12,113 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:12,113 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:12,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:12,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:12,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:12,115 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,115 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:12,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:12,115 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,115 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:12,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:12,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:12,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:12,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:12,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:12,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:12,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4636f510[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:12,125 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,125 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:12,125 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,125 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:12,125 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:12,125 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:12,128 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:12,128 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:12,128 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:12,128 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:12,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:12,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:12,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:12,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:12,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:12,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:12,131 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:12,132 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:12,132 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:12,132 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:12,132 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:12,133 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,133 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:12,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:12,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,133 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:12,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:12,134 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:12,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:12,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:12,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:12,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:12,135 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:12,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:12,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:12,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:12,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:12,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:12,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:12,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:12,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:12,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:12,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,137 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:12,137 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:12,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:12,138 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:12,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,138 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,138 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:12,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:12,183 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:12,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567772181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,185 DEBUG [Thread-2458] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., hostname=jenkins-hbase4.apache.org,40885,1682567618305, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 03:55:12,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:12,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567772183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,191 DEBUG [Thread-2450] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., hostname=jenkins-hbase4.apache.org,40885,1682567618305, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 03:55:12,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:12,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59744 deadline: 1682567772195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,204 DEBUG [Thread-2456] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4174 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., hostname=jenkins-hbase4.apache.org,40885,1682567618305, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 03:55:12,238 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:12,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:12,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:12,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:12,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:12,241 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:12,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:12,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:12,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:12,242 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,242 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:12,243 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,243 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:12,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:12,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:12,243 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,243 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:12,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:12,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:12,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:12,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:12,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:12,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:12,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:12,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:12,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:12,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:12,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:12,244 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:12,244 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:12,244 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:12,245 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:12,245 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:12,245 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:12,245 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,245 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:12,245 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:12,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:12,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:12,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:12,248 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:12,248 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,248 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:12,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:12,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:12,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:12,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e5e80dc[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:12,249 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:12,249 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,250 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,250 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:12,250 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:12,250 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:12,250 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,250 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:12,252 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,252 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:12,252 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:12,252 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:12,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:12,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:12,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:12,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:12,254 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:12,254 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:12,254 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:12,254 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:12,254 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:12,256 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,256 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,256 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:12,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:12,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:12,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,256 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:12,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:12,257 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,261 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:12,261 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:12,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:12,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:12,261 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:12,261 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:12,261 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:12,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:12,261 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:12,261 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:12,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:12,261 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,261 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:12,262 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,262 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,262 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:12,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:12,362 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:12,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:12,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:12,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:12,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:12,365 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:12,365 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:12,365 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:12,365 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:12,365 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,365 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:12,366 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:12,366 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:12,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:12,367 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,367 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:12,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:12,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:12,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:12,368 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:12,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:12,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:12,368 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:12,368 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:12,368 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:12,368 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:12,368 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:12,368 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:12,368 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:12,368 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:12,368 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:12,369 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:12,369 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:12,369 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,369 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:12,369 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:12,370 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,370 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:12,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:12,370 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,370 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:12,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:12,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:12,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:12,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:12,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:12,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@773b898e[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:12,372 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:12,372 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,373 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:12,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:12,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:12,373 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,373 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:12,376 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:12,376 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:12,376 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,376 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:12,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:12,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:12,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:12,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:12,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:12,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:12,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:12,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:12,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:12,378 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:12,378 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:12,378 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:12,378 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:12,378 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:12,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:12,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:12,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:12,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:12,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:12,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:12,380 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:12,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:12,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:12,381 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,383 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,383 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:12,383 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:12,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:12,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:12,384 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:12,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:12,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:12,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:12,384 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:12,384 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:12,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:12,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:12,384 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,384 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:12,384 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:12,384 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,384 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:12,384 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:12,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59742 deadline: 1682567772383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,401 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=169, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/c97bf34fdaaa4392a1a59d1aee81aecc 2023-04-27 03:55:12,412 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/5bd8c687fcbd42cbbc8098fc66db6f34 2023-04-27 03:55:12,423 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/ee74c2f3e74d4df5ac7d2d0ab888840c 2023-04-27 03:55:12,426 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/c97bf34fdaaa4392a1a59d1aee81aecc as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/c97bf34fdaaa4392a1a59d1aee81aecc 2023-04-27 03:55:12,429 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/c97bf34fdaaa4392a1a59d1aee81aecc, entries=150, sequenceid=169, filesize=25.3 K 2023-04-27 03:55:12,430 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/5bd8c687fcbd42cbbc8098fc66db6f34 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/5bd8c687fcbd42cbbc8098fc66db6f34 2023-04-27 03:55:12,432 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/5bd8c687fcbd42cbbc8098fc66db6f34, entries=100, sequenceid=169, filesize=9.4 K 2023-04-27 03:55:12,433 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/ee74c2f3e74d4df5ac7d2d0ab888840c as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/ee74c2f3e74d4df5ac7d2d0ab888840c 2023-04-27 03:55:12,436 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/ee74c2f3e74d4df5ac7d2d0ab888840c, entries=100, sequenceid=169, filesize=9.4 K 2023-04-27 03:55:12,436 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for e0314b11a1068d19a1052690701d1053 in 857ms, sequenceid=169, compaction requested=true 2023-04-27 03:55:12,437 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:12,437 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:12,437 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:12,437 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:12,437 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:12,437 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:12,438 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77892 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:12,438 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33922 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:12,438 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): e0314b11a1068d19a1052690701d1053/A is initiating minor compaction (all files) 2023-04-27 03:55:12,438 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e0314b11a1068d19a1052690701d1053/B is initiating minor compaction (all files) 2023-04-27 03:55:12,438 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e0314b11a1068d19a1052690701d1053/A in TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:12,438 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e0314b11a1068d19a1052690701d1053/B in TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:12,438 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/35f8386ff0fb48f3b20c27cd2aec6d44, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/b7a7f1d9c640483d8e059408d51f2b52, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/c97bf34fdaaa4392a1a59d1aee81aecc] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp, totalSize=76.1 K 2023-04-27 03:55:12,438 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/68301c30cbe1434597c3b2e4345eba3f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/98ec8be65f11438d896ecca506d0a6ac, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/5bd8c687fcbd42cbbc8098fc66db6f34] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp, totalSize=33.1 K 2023-04-27 03:55:12,438 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 35f8386ff0fb48f3b20c27cd2aec6d44, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1682567709049 2023-04-27 03:55:12,438 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 68301c30cbe1434597c3b2e4345eba3f, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1682567709049 2023-04-27 03:55:12,438 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting b7a7f1d9c640483d8e059408d51f2b52, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1682567709762 2023-04-27 03:55:12,439 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 98ec8be65f11438d896ecca506d0a6ac, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1682567709762 2023-04-27 03:55:12,439 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting c97bf34fdaaa4392a1a59d1aee81aecc, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1682567710942 2023-04-27 03:55:12,439 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 5bd8c687fcbd42cbbc8098fc66db6f34, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1682567710948 2023-04-27 03:55:12,444 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e0314b11a1068d19a1052690701d1053#B#compaction#342 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:12,453 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): e0314b11a1068d19a1052690701d1053#A#compaction#343 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:12,484 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:12,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:12,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:12,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:12,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:12,487 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:12,487 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:12,487 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:12,487 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:12,488 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,488 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:12,489 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:12,489 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:12,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:12,490 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:12,490 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:12,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:12,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:12,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:12,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:12,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:12,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:12,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:12,491 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:12,491 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:12,491 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:12,491 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:12,491 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:12,491 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:12,491 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:12,491 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e0314b11a1068d19a1052690701d1053 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-27 03:55:12,492 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=A 2023-04-27 03:55:12,492 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:12,492 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=B 2023-04-27 03:55:12,492 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:12,492 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=C 2023-04-27 03:55:12,492 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:12,508 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304274d0c380bc8344d3f8e6d8eaf9afe0dad to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304274d0c380bc8344d3f8e6d8eaf9afe0dad 2023-04-27 03:55:12,854 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/5a461389fded4a16aa7fffb53f7bd96d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/5a461389fded4a16aa7fffb53f7bd96d 2023-04-27 03:55:12,857 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e0314b11a1068d19a1052690701d1053/B of e0314b11a1068d19a1052690701d1053 into 5a461389fded4a16aa7fffb53f7bd96d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:12,857 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:12,857 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., storeName=e0314b11a1068d19a1052690701d1053/B, priority=13, startTime=1682567712437; duration=0sec 2023-04-27 03:55:12,857 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:12,858 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:12,859 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33922 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:12,859 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e0314b11a1068d19a1052690701d1053/C is initiating minor compaction (all files) 2023-04-27 03:55:12,859 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e0314b11a1068d19a1052690701d1053/C in TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:12,859 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/7504b76793e44df8af079aec034474b5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/9726a89c9b7b49bd9e016efaf6d27da5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/ee74c2f3e74d4df5ac7d2d0ab888840c] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp, totalSize=33.1 K 2023-04-27 03:55:12,860 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 7504b76793e44df8af079aec034474b5, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1682567709049 2023-04-27 03:55:12,860 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 9726a89c9b7b49bd9e016efaf6d27da5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1682567709762 2023-04-27 03:55:12,860 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting ee74c2f3e74d4df5ac7d2d0ab888840c, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1682567710948 2023-04-27 03:55:12,867 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e0314b11a1068d19a1052690701d1053#C#compaction#345 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:12,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:12,890 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:12,913 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=188, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/8e9c91e83bcc4e6396588160181bd390 2023-04-27 03:55:12,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:12,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567772925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:13,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567773030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:13,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567773211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,214 DEBUG [Thread-2452] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., hostname=jenkins-hbase4.apache.org,40885,1682567618305, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 03:55:13,235 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:13,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567773234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,280 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/a802f0aea21b4c809edbc1442886670f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/a802f0aea21b4c809edbc1442886670f 2023-04-27 03:55:13,284 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e0314b11a1068d19a1052690701d1053/C of e0314b11a1068d19a1052690701d1053 into a802f0aea21b4c809edbc1442886670f(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:13,284 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:13,284 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., storeName=e0314b11a1068d19a1052690701d1053/C, priority=13, startTime=1682567712437; duration=0sec 2023-04-27 03:55:13,284 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:13,330 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=188 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/efc3a3fda3b742918bb8153d78279716 2023-04-27 03:55:13,340 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=188 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/be45cda04b91443abe48dba1667a81bd 2023-04-27 03:55:13,345 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/8e9c91e83bcc4e6396588160181bd390 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/8e9c91e83bcc4e6396588160181bd390 2023-04-27 03:55:13,348 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/8e9c91e83bcc4e6396588160181bd390, entries=150, sequenceid=188, filesize=25.3 K 2023-04-27 03:55:13,349 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/efc3a3fda3b742918bb8153d78279716 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/efc3a3fda3b742918bb8153d78279716 2023-04-27 03:55:13,353 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/efc3a3fda3b742918bb8153d78279716, entries=150, sequenceid=188, filesize=11.8 K 2023-04-27 03:55:13,353 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/be45cda04b91443abe48dba1667a81bd as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/be45cda04b91443abe48dba1667a81bd 2023-04-27 03:55:13,357 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/be45cda04b91443abe48dba1667a81bd, entries=150, sequenceid=188, filesize=11.8 K 2023-04-27 03:55:13,357 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for e0314b11a1068d19a1052690701d1053 in 866ms, sequenceid=188, compaction requested=false 2023-04-27 03:55:13,358 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:13,358 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:13,358 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:13,358 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:13,358 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,358 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:13,358 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:13,361 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,361 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:13,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:13,362 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,362 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:13,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:13,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:13,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:13,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:13,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7900d70b[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:13,363 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:13,363 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,364 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,364 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:13,364 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:13,365 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:13,365 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,365 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:13,369 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:13,369 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:13,369 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:13,369 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:13,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:13,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:13,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:13,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:13,371 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:13,371 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:13,371 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:13,371 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:13,371 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:13,375 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,375 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,375 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:13,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:13,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:13,376 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:13,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:13,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,378 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:13,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:13,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:13,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:13,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:13,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:13,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:13,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:13,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:13,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:13,380 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:13,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,380 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:13,381 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:13,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:13,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:13,481 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:13,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:13,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:13,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:13,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:13,484 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:13,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:13,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:13,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:13,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:13,485 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:13,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:13,485 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:13,486 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,486 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:13,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:13,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:13,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:13,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:13,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:13,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:13,487 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:13,487 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:13,487 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:13,487 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:13,487 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:13,487 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:13,487 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:13,487 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e0314b11a1068d19a1052690701d1053 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-27 03:55:13,487 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=A 2023-04-27 03:55:13,488 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:13,488 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=B 2023-04-27 03:55:13,488 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:13,488 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=C 2023-04-27 03:55:13,488 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:13,501 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427805ff193ca52451ab188635e2ba5ebee to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427805ff193ca52451ab188635e2ba5ebee 2023-04-27 03:55:13,506 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=203, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/2cb4e0250cf84600931b7c72471d3c50 2023-04-27 03:55:13,516 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/e9098b676caf4e54a74b379073818371 2023-04-27 03:55:13,525 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/9b04c104c01144cca4cc77b931c5fa2c 2023-04-27 03:55:13,528 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/2cb4e0250cf84600931b7c72471d3c50 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/2cb4e0250cf84600931b7c72471d3c50 2023-04-27 03:55:13,531 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/2cb4e0250cf84600931b7c72471d3c50, entries=150, sequenceid=203, filesize=25.3 K 2023-04-27 03:55:13,532 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/e9098b676caf4e54a74b379073818371 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/e9098b676caf4e54a74b379073818371 2023-04-27 03:55:13,535 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/e9098b676caf4e54a74b379073818371, entries=150, sequenceid=203, filesize=11.8 K 2023-04-27 03:55:13,536 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/9b04c104c01144cca4cc77b931c5fa2c as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/9b04c104c01144cca4cc77b931c5fa2c 2023-04-27 03:55:13,542 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/9b04c104c01144cca4cc77b931c5fa2c, entries=150, sequenceid=203, filesize=11.8 K 2023-04-27 03:55:13,542 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for e0314b11a1068d19a1052690701d1053 in 55ms, sequenceid=203, compaction requested=true 2023-04-27 03:55:13,542 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:13,542 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:13,542 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:13,543 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:13,543 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,543 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:13,543 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:13,544 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,544 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:13,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:13,545 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,545 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:13,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:13,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:13,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:13,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:13,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@86f334e[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:13,546 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:13,546 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,548 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,548 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:13,548 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:13,548 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,549 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:13,548 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:13,550 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,550 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:13,550 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:13,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:13,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:13,550 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:13,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:13,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:13,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:13,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:13,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:13,553 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:13,553 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:13,553 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:13,553 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:13,553 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:13,554 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,554 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:13,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:13,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,554 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:13,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:13,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:13,555 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:13,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:13,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,556 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:13,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,558 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:13,558 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,558 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:13,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:13,558 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:13,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:13,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:13,558 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:13,559 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,559 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,559 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,559 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,559 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:13,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:13,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:13,558 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:13,559 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:13,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:13,659 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:13,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:13,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:13,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:13,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:13,665 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:13,665 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:13,666 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:13,666 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:13,666 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,666 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:13,668 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:13,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:13,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:13,668 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:13,669 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,669 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:13,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:13,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:13,670 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:13,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:13,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:13,670 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:13,670 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:13,670 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:13,670 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:13,670 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:13,670 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:13,670 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:13,671 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e0314b11a1068d19a1052690701d1053 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-27 03:55:13,671 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=A 2023-04-27 03:55:13,671 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:13,671 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=B 2023-04-27 03:55:13,671 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:13,671 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=C 2023-04-27 03:55:13,671 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:13,674 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/a93bdbf438fe472ca751f3e1d414fa38 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/a93bdbf438fe472ca751f3e1d414fa38 2023-04-27 03:55:13,681 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e0314b11a1068d19a1052690701d1053/A of e0314b11a1068d19a1052690701d1053 into a93bdbf438fe472ca751f3e1d414fa38(size=25.7 K), total size for store is 76.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-27 03:55:13,681 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:13,681 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., storeName=e0314b11a1068d19a1052690701d1053/A, priority=13, startTime=1682567712437; duration=1sec 2023-04-27 03:55:13,681 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:13,683 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427a0b70bb953d7443eacd72dea645403e4 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427a0b70bb953d7443eacd72dea645403e4 2023-04-27 03:55:13,688 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=209, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/11b002c77c7a4fc18ad536615b61ad68 2023-04-27 03:55:13,696 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/9f165d5d1e1145ed85db124fdbbe8aef 2023-04-27 03:55:13,705 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/0a40116ccc1c4f13819167717e5ad5e6 2023-04-27 03:55:13,708 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/11b002c77c7a4fc18ad536615b61ad68 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/11b002c77c7a4fc18ad536615b61ad68 2023-04-27 03:55:13,711 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/11b002c77c7a4fc18ad536615b61ad68, entries=100, sequenceid=209, filesize=18.5 K 2023-04-27 03:55:13,712 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/9f165d5d1e1145ed85db124fdbbe8aef as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/9f165d5d1e1145ed85db124fdbbe8aef 2023-04-27 03:55:13,715 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/9f165d5d1e1145ed85db124fdbbe8aef, entries=50, sequenceid=209, filesize=7.1 K 2023-04-27 03:55:13,716 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/0a40116ccc1c4f13819167717e5ad5e6 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/0a40116ccc1c4f13819167717e5ad5e6 2023-04-27 03:55:13,719 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/0a40116ccc1c4f13819167717e5ad5e6, entries=50, sequenceid=209, filesize=7.1 K 2023-04-27 03:55:13,720 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=6.71 KB/6870 for e0314b11a1068d19a1052690701d1053 in 50ms, sequenceid=209, compaction requested=true 2023-04-27 03:55:13,720 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:13,720 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:13,720 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:13,720 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:13,720 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,720 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:13,720 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:13,721 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:13,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:13,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:13,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:13,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:13,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:13,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:13,723 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:13,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bd1aff[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:13,723 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,725 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,725 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:13,725 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:13,725 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:13,725 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,725 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:13,726 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:13,726 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:13,726 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,726 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:13,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:13,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:13,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:13,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:13,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:13,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:13,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:13,729 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:13,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:13,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:13,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:13,729 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:13,731 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,731 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,731 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:13,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:13,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:13,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:13,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:13,732 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:13,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,733 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,735 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,735 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:13,735 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:13,735 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:13,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:13,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:13,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:13,735 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:13,735 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:13,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:13,735 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:13,735 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,735 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:13,736 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,736 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:13,736 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:13,810 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e0314b11a1068d19a1052690701d1053 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:55:13,810 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=A 2023-04-27 03:55:13,810 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:13,810 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=B 2023-04-27 03:55:13,810 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:13,810 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=C 2023-04-27 03:55:13,810 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:13,836 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:13,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:13,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:13,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:13,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:13,838 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:13,838 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:13,839 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:13,839 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:13,839 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,839 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:13,840 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:13,840 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:13,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:13,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:13,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:13,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:13,841 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:13,842 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:13,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:13,842 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,842 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:13,842 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:13,842 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:13,842 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:13,842 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:13,842 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:13,842 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:13,842 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:13,842 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:13,842 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:13,843 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:13,843 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:13,843 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,843 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:13,843 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:13,844 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,844 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:13,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:13,844 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,844 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:13,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:13,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:13,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:13,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:13,846 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:13,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@296194df[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:13,846 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,847 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:13,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:13,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:13,848 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,848 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:13,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:13,849 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:13,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:13,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:13,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:13,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:13,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:13,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:13,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:13,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:13,852 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:13,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:13,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:13,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:13,852 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:13,853 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:13,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:13,853 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,854 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:13,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:13,854 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:13,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:13,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:13,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,855 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:13,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,863 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:13,863 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:13,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:13,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:13,863 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,863 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:13,863 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:13,863 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:13,863 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:13,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:13,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:13,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:13,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,864 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:13,864 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:13,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567713906, max timestamp=1682567713961Num uniques 859; 2023-04-27 03:55:13,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567713906, max timestamp=1682567713961Num uniques 859; 2023-04-27 03:55:13,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:55:13,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:55:13,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567713906, max timestamp=1682567713961Num uniques 859; 2023-04-27 03:55:13,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:55:13,964 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:13,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:13,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:13,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:13,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:13,967 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:13,967 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:13,967 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:13,967 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:13,968 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,968 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:13,969 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,969 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:13,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:13,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:13,969 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,969 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:13,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:13,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:13,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:13,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:13,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:13,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:13,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:13,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:13,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:13,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:13,971 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:13,971 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:13,971 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:13,971 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:13,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:13,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:13,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:13,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:13,973 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:13,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:13,973 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,974 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:13,974 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:13,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:13,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:13,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:13,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56a6b677[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:13,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:13,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,977 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,977 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:13,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:13,977 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:13,978 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:13,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:13,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:13,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:13,979 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:13,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:13,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:13,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:13,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:13,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:13,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:13,981 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:13,981 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:13,981 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:13,981 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:13,981 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:13,983 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:13,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:13,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:13,983 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:13,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:13,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:13,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,986 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,986 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:13,986 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:13,986 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:13,986 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,987 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:13,987 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,987 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:13,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:13,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:13,987 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:13,987 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:13,988 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,988 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:13,988 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:13,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:13,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:13,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:13,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:13,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:13,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:14,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:14,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567774069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,087 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:14,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:14,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:14,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:14,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:14,090 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:14,090 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:14,090 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:14,090 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:14,090 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,090 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,093 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,093 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:14,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:14,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:14,093 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,093 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,094 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:14,094 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:14,094 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:14,094 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:14,094 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:14,094 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:14,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:14,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:14,095 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:14,095 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:14,095 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:14,095 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:14,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:14,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:14,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:14,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:14,096 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:14,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:14,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:14,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:14,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,097 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,098 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,098 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:14,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:14,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:14,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27a21aae[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:14,098 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:14,098 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,099 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,099 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:14,100 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:14,100 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,100 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:14,100 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:14,101 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,101 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:14,101 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:14,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:14,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:14,101 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:14,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:14,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:14,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:14,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:14,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:14,103 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:14,103 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:14,103 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:14,103 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:14,103 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:14,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:14,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:14,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:14,111 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:14,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:14,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:14,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:14,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,112 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:14,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,114 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,114 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,114 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,114 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,114 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,114 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,114 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,114 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:14,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:14,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:14,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:14,114 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:14,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:14,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:14,115 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:14,115 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:14,114 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:14,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:14,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:14,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:14,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567774181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,215 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:14,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:14,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:14,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:14,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:14,218 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:14,218 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:14,218 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:14,218 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:14,218 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,218 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,219 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,219 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:14,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:14,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:14,220 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,220 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:14,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:14,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:14,221 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:14,221 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:14,221 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:14,221 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:14,221 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:14,221 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:14,221 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:14,222 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:14,222 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:14,222 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:14,222 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:14,222 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:14,222 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:14,222 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,222 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:14,222 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:14,223 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,223 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:14,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:14,224 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,224 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:14,224 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304272251a8a1bc1a47f38d1c3313e9bb8b00 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272251a8a1bc1a47f38d1c3313e9bb8b00 2023-04-27 03:55:14,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:14,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:14,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:14,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:14,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d229647[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:14,231 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:14,231 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,236 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:14,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:14,237 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:14,237 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,237 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:14,238 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,238 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:14,238 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:14,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:14,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:14,238 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:14,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:14,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:14,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:14,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:14,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:14,240 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:14,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:14,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:14,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:14,241 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:14,242 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,242 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,242 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:14,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:14,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:14,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:14,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:14,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:14,243 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:14,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,244 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:14,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,250 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,250 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,250 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:14,250 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:14,250 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:14,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:14,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:14,250 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,251 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,251 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,251 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,251 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,251 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:14,251 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:14,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:14,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:14,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:14,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:14,351 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:14,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:14,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:14,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:14,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:14,354 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:14,354 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:14,354 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:14,354 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:14,354 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,355 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,356 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:14,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:14,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:14,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:14,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:14,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:14,357 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:14,357 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:14,357 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:14,357 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:14,357 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:14,357 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:14,358 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:14,358 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:14,358 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:14,358 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:14,358 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:14,358 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:14,358 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:14,358 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,358 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:14,358 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:14,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:14,360 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:14,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:14,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:14,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:14,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:14,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:14,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68402450[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:14,361 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:14,361 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,363 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,364 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,364 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:14,364 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:14,364 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:14,364 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:14,365 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,365 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:14,365 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:14,365 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:14,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:14,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:14,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:14,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:14,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:14,367 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:14,368 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:14,368 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:14,368 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:14,368 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:14,369 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,369 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,369 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:14,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:14,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:14,369 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:14,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:14,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:14,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:14,370 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,372 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,372 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,372 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,372 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:14,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:14,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:14,372 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,373 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,373 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,373 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:14,373 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:14,372 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:14,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:14,375 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:14,375 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:14,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:14,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:14,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:14,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:14,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:14,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567774388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,473 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:14,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:14,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:14,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:14,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:14,476 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:14,476 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:14,476 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:14,476 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:14,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,479 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:14,479 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:14,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:14,480 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,480 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:14,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:14,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:14,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:14,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:14,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:14,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:14,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:14,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:14,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:14,481 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:14,481 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:14,481 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:14,481 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:14,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:14,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:14,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:14,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:14,483 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,483 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:14,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:14,483 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,483 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:14,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:14,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:14,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:14,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:14,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:14,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bb63221[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:14,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,486 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,486 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:14,486 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:14,486 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,486 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:14,486 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:14,487 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,487 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:14,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:14,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:14,487 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:14,488 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:14,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:14,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:14,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:14,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:14,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:14,490 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:14,490 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:14,490 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:14,490 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:14,490 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:14,492 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,492 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:14,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:14,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,492 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:14,492 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:14,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:14,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:14,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:14,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,493 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:14,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,497 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:14,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:14,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:14,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:14,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:14,497 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:14,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:14,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:14,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:14,497 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:14,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:14,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:14,598 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:14,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:14,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:14,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:14,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:14,601 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:14,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:14,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:14,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:14,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,603 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:14,603 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:14,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:14,603 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,603 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:14,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:14,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:14,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:14,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:14,604 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:14,604 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:14,604 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:14,604 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:14,606 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:14,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:14,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,607 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:14,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:14,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:14,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:14,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:14,608 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:14,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cf63e93[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:14,609 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,610 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:14,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:14,610 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,611 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:14,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:14,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,612 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:14,612 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:14,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:14,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:14,612 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:14,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:14,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:14,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:14,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:14,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:14,614 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:14,614 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:14,614 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:14,614 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:14,614 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:14,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:14,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:14,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:14,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:14,616 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:14,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:14,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:14,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,619 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:14,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,621 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:14,621 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:14,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:14,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:14,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:14,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:14,621 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,621 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,621 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:14,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:14,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,622 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:14,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:14,622 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:14,622 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:14,647 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=220, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/bc9f6bd12a4e408a86a169eae71802bf 2023-04-27 03:55:14,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:14,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567774695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,722 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:14,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:14,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:14,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:14,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:14,725 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:14,725 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:14,725 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:14,725 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:14,726 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,726 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,730 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:14,730 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:14,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:14,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:14,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:14,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,731 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:14,732 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:14,732 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:14,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:14,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:14,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:14,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:14,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:14,732 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:14,732 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:14,732 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:14,732 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:14,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:14,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:14,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:14,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:14,734 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,734 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:14,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:14,734 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,734 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:14,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:14,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:14,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:14,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:14,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65bf17e4[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:14,735 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:14,735 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,737 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,737 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:14,737 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:14,737 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:14,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:14,741 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:14,741 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,741 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:14,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:14,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:14,741 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:14,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:14,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:14,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:14,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:14,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:14,743 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:14,743 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:14,743 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:14,743 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:14,743 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:14,744 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,745 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:14,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,745 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:14,745 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:14,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,746 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:14,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,748 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,748 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:14,748 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,748 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:14,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:14,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:14,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,749 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:14,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:14,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:14,749 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:14,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,749 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:14,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:14,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:14,849 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:14,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:14,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:14,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:14,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:14,852 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:14,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:14,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:14,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:14,853 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,853 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,854 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,854 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:14,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:14,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:14,854 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,854 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:14,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:14,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:14,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:14,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:14,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:14,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:14,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:14,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:14,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:14,856 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:14,856 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:14,856 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:14,856 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:14,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:14,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:14,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:14,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:14,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,859 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:14,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:14,859 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,859 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:14,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:14,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:14,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:14,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:14,861 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:14,861 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cb4dda7[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:14,862 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:14,863 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:14,863 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,863 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:14,863 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:14,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,864 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:14,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:14,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:14,864 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:14,864 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:14,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:14,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:14,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:14,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:14,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:14,866 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:14,866 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:14,866 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:14,866 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:14,866 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:14,868 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,868 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:14,868 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:14,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:14,868 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:14,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:14,869 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:14,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,872 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,872 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,872 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:14,872 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:14,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:14,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:14,872 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,872 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:14,872 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,873 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,873 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,873 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:14,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:14,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:14,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:14,873 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:14,873 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:14,973 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:14,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:14,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:14,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:14,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:14,976 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:14,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:14,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:14,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:14,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,978 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,978 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:14,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:14,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:14,979 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,979 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:14,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:14,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:14,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:14,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:14,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:14,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:14,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:14,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:14,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:14,980 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:14,980 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:14,980 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:14,980 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:14,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:14,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:14,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:14,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:14,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,982 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:14,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:14,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:14,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:14,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:14,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:14,983 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:14,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36a9c12a[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:14,983 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,985 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:14,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:14,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:14,985 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,985 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:14,987 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:14,987 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:14,987 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:14,987 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:14,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:14,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:14,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:14,989 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:14,989 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:14,989 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:14,989 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:14,989 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:14,991 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,991 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:14,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,991 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:14,991 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,993 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:14,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,995 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,995 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,995 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:14,995 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:14,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:14,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:14,995 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:14,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:14,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:14,995 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:14,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:14,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:14,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:14,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:14,996 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:14,996 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:14,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,056 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/36df2ac31305402995405d4bd23f948f 2023-04-27 03:55:15,096 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:15,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:15,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:15,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:15,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:15,099 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:15,099 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:15,099 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:15,099 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:15,099 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,099 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:15,101 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:15,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:15,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:15,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:15,101 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,102 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,102 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:15,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:15,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:15,102 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:15,102 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:15,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:15,102 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:15,102 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:15,102 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:15,102 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:15,102 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:15,102 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:15,102 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:15,103 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:15,103 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:15,103 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:15,103 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:15,103 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,103 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:15,103 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:15,105 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,106 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:15,106 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,106 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:15,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:15,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:15,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ba663ca[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:15,108 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:15,108 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,112 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,112 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:15,112 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:15,112 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:15,112 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,112 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:15,113 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,113 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:15,113 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:15,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:15,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:15,113 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:15,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:15,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:15,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:15,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:15,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:15,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:15,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:15,115 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:15,116 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:15,116 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:15,116 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:15,116 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:15,117 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,117 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:15,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,117 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:15,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:15,119 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:15,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,123 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:15,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:15,123 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:15,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:15,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:15,123 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:15,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:15,123 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:15,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:15,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:15,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,123 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:15,124 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:15,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:15,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:15,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:15,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567775205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,224 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:15,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:15,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:15,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:15,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:15,227 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:15,227 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:15,227 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:15,227 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:15,228 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,228 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:15,229 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,229 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:15,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:15,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:15,229 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,229 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:15,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:15,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:15,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:15,230 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:15,230 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:15,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:15,230 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:15,230 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:15,230 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:15,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:15,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:15,231 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:15,231 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:15,231 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:15,231 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:15,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:15,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:15,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:15,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:15,232 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,232 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:15,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:15,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:15,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,233 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:15,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:15,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:15,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:15,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:15,234 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:15,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@263b536d[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:15,234 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,235 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,235 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:15,235 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,235 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:15,235 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:15,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:15,238 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,238 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:15,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:15,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:15,238 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:15,238 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:15,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:15,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:15,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:15,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:15,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:15,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:15,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:15,240 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:15,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:15,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:15,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:15,241 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:15,242 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,242 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,242 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:15,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:15,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:15,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:15,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:15,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:15,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,243 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:15,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:15,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:15,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,244 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:15,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:15,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,246 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,246 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:15,246 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:15,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:15,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:15,246 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:15,246 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:15,246 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:15,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:15,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:15,246 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:15,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:15,247 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:15,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:15,247 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:15,347 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:15,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:15,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:15,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:15,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:15,350 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:15,350 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:15,350 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:15,350 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:15,350 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,350 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:15,352 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,352 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:15,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:15,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:15,352 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,352 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:15,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:15,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:15,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:15,353 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:15,353 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:15,353 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:15,353 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:15,353 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:15,353 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:15,353 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:15,353 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:15,353 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:15,353 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:15,353 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:15,354 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:15,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:15,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:15,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:15,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:15,357 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,357 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:15,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:15,357 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,357 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:15,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:15,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:15,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:15,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:15,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:15,358 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:15,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cc8fbe8[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:15,358 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,359 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:15,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:15,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:15,360 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,360 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:15,361 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:15,361 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:15,361 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,361 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:15,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:15,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:15,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:15,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:15,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:15,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:15,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:15,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:15,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:15,364 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:15,364 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:15,364 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:15,364 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:15,364 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:15,365 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,365 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,365 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:15,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:15,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:15,366 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:15,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:15,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:15,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:15,368 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:15,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:15,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:15,371 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,371 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:15,371 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:15,371 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,371 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:15,371 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:15,371 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:15,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:15,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:15,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:15,371 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,371 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,371 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:15,371 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:15,371 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:15,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:15,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:15,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:15,466 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/5973df858e06420c842f3492dab2bcb9 2023-04-27 03:55:15,470 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/bc9f6bd12a4e408a86a169eae71802bf as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/bc9f6bd12a4e408a86a169eae71802bf 2023-04-27 03:55:15,471 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:15,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:15,473 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/bc9f6bd12a4e408a86a169eae71802bf, entries=200, sequenceid=220, filesize=32.1 K 2023-04-27 03:55:15,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:15,474 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/36df2ac31305402995405d4bd23f948f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/36df2ac31305402995405d4bd23f948f 2023-04-27 03:55:15,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:15,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:15,475 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:15,475 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:15,475 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:15,475 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:15,475 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,475 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:15,476 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:15,476 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:15,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:15,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:15,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:15,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:15,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:15,477 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:15,478 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/36df2ac31305402995405d4bd23f948f, entries=150, sequenceid=220, filesize=11.8 K 2023-04-27 03:55:15,478 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:15,478 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:15,478 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:15,478 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:15,478 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:15,478 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:15,478 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:15,478 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:15,478 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/5973df858e06420c842f3492dab2bcb9 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/5973df858e06420c842f3492dab2bcb9 2023-04-27 03:55:15,478 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:15,479 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:15,479 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:15,479 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:15,479 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:15,479 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,479 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:15,479 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:15,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,480 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:15,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:15,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:15,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:15,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:15,481 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/5973df858e06420c842f3492dab2bcb9, entries=150, sequenceid=220, filesize=11.8 K 2023-04-27 03:55:15,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:15,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,482 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e0314b11a1068d19a1052690701d1053 in 1672ms, sequenceid=220, compaction requested=true 2023-04-27 03:55:15,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:15,482 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:15,482 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:15,482 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 03:55:15,482 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 03:55:15,482 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:15,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:15,483 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:15,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7410cbca[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:15,483 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:15,483 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,484 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 55804 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 03:55:15,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,484 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e0314b11a1068d19a1052690701d1053/B is initiating minor compaction (all files) 2023-04-27 03:55:15,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,484 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e0314b11a1068d19a1052690701d1053/B in TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:15,484 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:15,484 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:15,484 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 129880 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 03:55:15,484 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:15,484 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): e0314b11a1068d19a1052690701d1053/A is initiating minor compaction (all files) 2023-04-27 03:55:15,484 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/5a461389fded4a16aa7fffb53f7bd96d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/efc3a3fda3b742918bb8153d78279716, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/e9098b676caf4e54a74b379073818371, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/9f165d5d1e1145ed85db124fdbbe8aef, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/36df2ac31305402995405d4bd23f948f] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp, totalSize=54.5 K 2023-04-27 03:55:15,484 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e0314b11a1068d19a1052690701d1053/A in TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:15,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,485 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/a93bdbf438fe472ca751f3e1d414fa38, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/8e9c91e83bcc4e6396588160181bd390, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/2cb4e0250cf84600931b7c72471d3c50, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/11b002c77c7a4fc18ad536615b61ad68, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/bc9f6bd12a4e408a86a169eae71802bf] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp, totalSize=126.8 K 2023-04-27 03:55:15,485 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:15,485 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 5a461389fded4a16aa7fffb53f7bd96d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1682567709774 2023-04-27 03:55:15,485 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting a93bdbf438fe472ca751f3e1d414fa38, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1682567709774 2023-04-27 03:55:15,485 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting efc3a3fda3b742918bb8153d78279716, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=188, earliestPutTs=1682567711719 2023-04-27 03:55:15,485 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 8e9c91e83bcc4e6396588160181bd390, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=188, earliestPutTs=1682567711719 2023-04-27 03:55:15,486 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting e9098b676caf4e54a74b379073818371, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1682567712913 2023-04-27 03:55:15,486 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 2cb4e0250cf84600931b7c72471d3c50, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1682567712913 2023-04-27 03:55:15,486 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 9f165d5d1e1145ed85db124fdbbe8aef, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1682567713664 2023-04-27 03:55:15,486 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 11b002c77c7a4fc18ad536615b61ad68, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1682567713624 2023-04-27 03:55:15,486 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 36df2ac31305402995405d4bd23f948f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1682567713777 2023-04-27 03:55:15,486 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting bc9f6bd12a4e408a86a169eae71802bf, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1682567713709 2023-04-27 03:55:15,489 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:15,489 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:15,489 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:15,489 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:15,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:15,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:15,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:15,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:15,492 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:15,492 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:15,492 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:15,492 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:15,492 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:15,496 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e0314b11a1068d19a1052690701d1053#B#compaction#357 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:15,496 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:15,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:15,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:15,498 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:15,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,499 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,499 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): e0314b11a1068d19a1052690701d1053#A#compaction#358 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:15,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:15,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:15,501 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:15,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:15,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:15,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:15,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:15,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,502 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:15,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:15,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:15,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:15,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:15,502 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:15,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:15,502 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:15,502 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:15,509 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/5c484af2a41846ab84b1d29789c81fda as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/5c484af2a41846ab84b1d29789c81fda 2023-04-27 03:55:15,512 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e0314b11a1068d19a1052690701d1053/B of e0314b11a1068d19a1052690701d1053 into 5c484af2a41846ab84b1d29789c81fda(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:15,512 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:15,512 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., storeName=e0314b11a1068d19a1052690701d1053/B, priority=11, startTime=1682567715482; duration=0sec 2023-04-27 03:55:15,513 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:15,513 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 03:55:15,514 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 55804 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 03:55:15,514 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e0314b11a1068d19a1052690701d1053/C is initiating minor compaction (all files) 2023-04-27 03:55:15,514 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e0314b11a1068d19a1052690701d1053/C in TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:15,514 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/a802f0aea21b4c809edbc1442886670f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/be45cda04b91443abe48dba1667a81bd, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/9b04c104c01144cca4cc77b931c5fa2c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/0a40116ccc1c4f13819167717e5ad5e6, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/5973df858e06420c842f3492dab2bcb9] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp, totalSize=54.5 K 2023-04-27 03:55:15,514 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting a802f0aea21b4c809edbc1442886670f, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1682567709774 2023-04-27 03:55:15,515 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting be45cda04b91443abe48dba1667a81bd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=188, earliestPutTs=1682567711719 2023-04-27 03:55:15,515 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 9b04c104c01144cca4cc77b931c5fa2c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1682567712913 2023-04-27 03:55:15,515 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 0a40116ccc1c4f13819167717e5ad5e6, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1682567713664 2023-04-27 03:55:15,516 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 5973df858e06420c842f3492dab2bcb9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1682567713777 2023-04-27 03:55:15,523 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e0314b11a1068d19a1052690701d1053#C#compaction#359 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:15,529 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/9393dfd093c2403bb8551e16913defb4 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/9393dfd093c2403bb8551e16913defb4 2023-04-27 03:55:15,533 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e0314b11a1068d19a1052690701d1053/C of e0314b11a1068d19a1052690701d1053 into 9393dfd093c2403bb8551e16913defb4(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:15,533 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:15,533 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., storeName=e0314b11a1068d19a1052690701d1053/C, priority=11, startTime=1682567715483; duration=0sec 2023-04-27 03:55:15,533 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:15,602 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:15,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:15,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:15,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:15,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:15,605 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:15,605 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:15,605 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:15,605 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:15,606 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,606 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:15,607 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:15,607 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:15,607 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:15,607 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:15,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:15,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:15,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:15,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:15,608 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:15,608 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:15,608 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:15,609 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:15,609 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:15,609 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:15,609 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:15,609 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e0314b11a1068d19a1052690701d1053 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-27 03:55:15,609 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=A 2023-04-27 03:55:15,609 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:15,609 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=B 2023-04-27 03:55:15,609 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:15,609 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=C 2023-04-27 03:55:15,609 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:15,919 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/f0ea729d72124b63b843973bdffb57f7 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/f0ea729d72124b63b843973bdffb57f7 2023-04-27 03:55:15,922 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e0314b11a1068d19a1052690701d1053/A of e0314b11a1068d19a1052690701d1053 into f0ea729d72124b63b843973bdffb57f7(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:15,922 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:15,923 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., storeName=e0314b11a1068d19a1052690701d1053/A, priority=11, startTime=1682567715482; duration=0sec 2023-04-27 03:55:15,923 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:16,023 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427abb555027a4b450da8906155d94f2dfb to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427abb555027a4b450da8906155d94f2dfb 2023-04-27 03:55:16,030 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=258, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/8a87a9b970be41c29990779fc018ff7b 2023-04-27 03:55:16,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:16,213 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:16,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:16,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567776226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:16,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567776229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:16,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59744 deadline: 1682567776232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:16,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567776232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,237 DEBUG [Thread-2456] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8207 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., hostname=jenkins-hbase4.apache.org,40885,1682567618305, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 03:55:16,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:16,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567776333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:16,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567776333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:16,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567776337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,441 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/7bde2bb886a1473baad9e39e919135e0 2023-04-27 03:55:16,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:16,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567776539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:16,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567776539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:16,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567776543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:16,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567776844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:16,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567776845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,851 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/7dfe0d7c9b7e4af68463de29784ef5fa 2023-04-27 03:55:16,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:16,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567776850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,854 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/8a87a9b970be41c29990779fc018ff7b as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/8a87a9b970be41c29990779fc018ff7b 2023-04-27 03:55:16,858 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/8a87a9b970be41c29990779fc018ff7b, entries=150, sequenceid=258, filesize=25.3 K 2023-04-27 03:55:16,859 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/7bde2bb886a1473baad9e39e919135e0 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/7bde2bb886a1473baad9e39e919135e0 2023-04-27 03:55:16,862 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/7bde2bb886a1473baad9e39e919135e0, entries=150, sequenceid=258, filesize=11.8 K 2023-04-27 03:55:16,863 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/7dfe0d7c9b7e4af68463de29784ef5fa as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/7dfe0d7c9b7e4af68463de29784ef5fa 2023-04-27 03:55:16,866 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/7dfe0d7c9b7e4af68463de29784ef5fa, entries=150, sequenceid=258, filesize=11.8 K 2023-04-27 03:55:16,866 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for e0314b11a1068d19a1052690701d1053 in 1257ms, sequenceid=258, compaction requested=false 2023-04-27 03:55:16,866 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:16,867 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:16,867 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:16,867 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:16,867 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,867 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:16,867 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:16,868 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,868 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:16,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:16,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:16,869 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:16,869 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:16,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:16,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:16,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:16,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:16,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:16,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ca04eb1[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:16,870 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:16,870 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:16,871 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:16,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:16,872 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:16,872 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:16,872 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,872 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:16,872 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:16,873 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,873 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:16,873 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:16,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:16,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:16,873 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:16,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:16,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:16,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:16,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:16,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:16,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:16,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:16,876 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:16,876 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:16,876 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:16,876 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:16,876 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:16,883 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:16,883 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:16,883 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:16,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:16,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:16,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:16,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:16,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:16,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:16,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:16,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:16,884 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:16,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:16,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:16,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:16,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,885 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:16,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:16,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,887 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,887 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:16,887 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:16,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:16,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:16,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:16,887 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:16,887 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:16,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:16,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:16,887 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:16,887 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:16,888 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:16,888 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,888 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:16,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:16,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:16,888 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:16,888 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:16,988 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:16,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:16,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:16,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:16,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:16,991 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:16,991 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:16,991 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:16,991 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:16,991 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:16,991 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:16,993 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:16,993 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:16,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:16,993 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:16,993 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:16,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:16,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:16,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:16,994 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:16,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:16,994 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:16,994 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:16,994 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:16,994 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:16,994 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:16,994 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:16,994 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:16,995 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e0314b11a1068d19a1052690701d1053 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:55:16,995 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=A 2023-04-27 03:55:16,995 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:16,995 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=B 2023-04-27 03:55:16,995 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:16,995 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=C 2023-04-27 03:55:16,995 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:17,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:17,230 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:17,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567717290, max timestamp=1682567717325Num uniques 859; 2023-04-27 03:55:17,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567717290, max timestamp=1682567717325Num uniques 859; 2023-04-27 03:55:17,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567717290, max timestamp=1682567717325Num uniques 859; 2023-04-27 03:55:17,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:55:17,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:55:17,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:55:17,406 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304271ebcc9b70b51476387480c319c898fdc to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304271ebcc9b70b51476387480c319c898fdc 2023-04-27 03:55:17,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9973, totalHeapSize=22912, min timestamp=1682567717372, max timestamp=1682567717393Num uniques 1018; 2023-04-27 03:55:17,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46613, new segment data size=9973, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-27 03:55:17,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9968, totalHeapSize=22912, min timestamp=1682567717372, max timestamp=1682567717393Num uniques 1018; 2023-04-27 03:55:17,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46608, new segment data size=9968, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-27 03:55:17,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9968, totalHeapSize=22912, min timestamp=1682567717372, max timestamp=1682567717393Num uniques 1018; 2023-04-27 03:55:17,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46608, new segment data size=9968, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-27 03:55:17,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:17,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567777447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:17,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567777448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:17,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567777448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:17,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567777449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:17,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567777556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:17,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567777556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:17,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567777556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:17,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567777556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:17,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567777760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:17,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567777760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:17,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567777760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:17,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567777761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,816 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=269, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/929f831c612a48b9b3a38b1a6bf21221 2023-04-27 03:55:17,825 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/13e87b1b6fa74a9f909ae442dd7d765f 2023-04-27 03:55:17,834 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/eee4ec52c36e42bbbfd777beb7db0c77 2023-04-27 03:55:17,837 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/929f831c612a48b9b3a38b1a6bf21221 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/929f831c612a48b9b3a38b1a6bf21221 2023-04-27 03:55:17,844 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/929f831c612a48b9b3a38b1a6bf21221, entries=150, sequenceid=269, filesize=25.4 K 2023-04-27 03:55:17,845 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/13e87b1b6fa74a9f909ae442dd7d765f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/13e87b1b6fa74a9f909ae442dd7d765f 2023-04-27 03:55:17,849 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/13e87b1b6fa74a9f909ae442dd7d765f, entries=150, sequenceid=269, filesize=11.9 K 2023-04-27 03:55:17,850 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/eee4ec52c36e42bbbfd777beb7db0c77 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/eee4ec52c36e42bbbfd777beb7db0c77 2023-04-27 03:55:17,853 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/eee4ec52c36e42bbbfd777beb7db0c77, entries=150, sequenceid=269, filesize=11.9 K 2023-04-27 03:55:17,853 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e0314b11a1068d19a1052690701d1053 in 859ms, sequenceid=269, compaction requested=true 2023-04-27 03:55:17,854 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:17,854 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:17,854 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:17,854 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:17,854 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,854 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:17,854 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:17,857 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,857 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:17,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:17,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:17,857 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:17,857 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:17,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:17,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:17,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:17,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:17,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:17,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cda73[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:17,859 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:17,859 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:17,860 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:17,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:17,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:17,860 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:17,861 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:17,861 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:17,861 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,861 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:17,862 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:17,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:17,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:17,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:17,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:17,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:17,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:17,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:17,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:17,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:17,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:17,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:17,864 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:17,864 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:17,864 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:17,864 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:17,864 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:17,866 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:17,866 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:17,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:17,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:17,867 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:17,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:17,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:17,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:17,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:17,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:17,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:17,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:17,868 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:17,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:17,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:17,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,869 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:17,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:17,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,871 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:17,871 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:17,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:17,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:17,871 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:17,871 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,871 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:17,871 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:17,871 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,871 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:17,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:17,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:17,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:17,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:17,872 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:17,872 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:17,872 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:17,872 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:17,972 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:17,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:17,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:17,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:17,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:17,975 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:17,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:17,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:17,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:17,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:17,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:17,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,977 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:17,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:17,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:17,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:17,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:17,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:17,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:17,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:17,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:17,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:17,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:17,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:17,978 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:17,978 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:17,978 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:17,978 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:17,978 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:17,979 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:17,979 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:17,979 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e0314b11a1068d19a1052690701d1053 3/3 column families, dataSize=134.18 KB heapSize=379.45 KB 2023-04-27 03:55:17,979 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=A 2023-04-27 03:55:17,979 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:17,979 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=B 2023-04-27 03:55:17,979 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:17,979 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=C 2023-04-27 03:55:17,979 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:17,994 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304275db9e5676d0045fa98fa73862b3bedb9 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304275db9e5676d0045fa98fa73862b3bedb9 2023-04-27 03:55:18,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:18,074 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:18,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:18,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567778077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:18,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567778082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,092 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:18,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567778083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:18,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567778084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:18,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567778186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:18,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567778193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:18,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567778193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:18,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567778194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:18,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567778394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,405 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=322, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/723578ff32b0494c9e92aa0bc7d42344 2023-04-27 03:55:18,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:18,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567778406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:18,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567778407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:18,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567778407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,418 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/6022e518344e48f7b5611b6700e12e99 2023-04-27 03:55:18,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:18,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567778703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:18,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567778715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:18,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567778715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:18,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567778716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,838 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/c8ba9c9a5cd14a0cb0b3f42dfa5b94b8 2023-04-27 03:55:18,842 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/723578ff32b0494c9e92aa0bc7d42344 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/723578ff32b0494c9e92aa0bc7d42344 2023-04-27 03:55:18,846 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/723578ff32b0494c9e92aa0bc7d42344, entries=150, sequenceid=322, filesize=25.4 K 2023-04-27 03:55:18,847 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/6022e518344e48f7b5611b6700e12e99 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/6022e518344e48f7b5611b6700e12e99 2023-04-27 03:55:18,852 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/6022e518344e48f7b5611b6700e12e99, entries=150, sequenceid=322, filesize=11.9 K 2023-04-27 03:55:18,853 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/c8ba9c9a5cd14a0cb0b3f42dfa5b94b8 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/c8ba9c9a5cd14a0cb0b3f42dfa5b94b8 2023-04-27 03:55:18,857 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/c8ba9c9a5cd14a0cb0b3f42dfa5b94b8, entries=150, sequenceid=322, filesize=11.9 K 2023-04-27 03:55:18,858 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~379.41 KB/388512, currentSize=46.96 KB/48090 for e0314b11a1068d19a1052690701d1053 in 879ms, sequenceid=322, compaction requested=true 2023-04-27 03:55:18,858 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:18,858 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:18,858 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:18,858 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:18,858 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,858 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:18,858 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:18,860 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,860 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:18,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:18,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:18,860 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:18,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:18,861 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:18,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:18,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:18,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:18,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:18,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b20762f[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:18,862 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:18,862 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:18,863 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,863 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:18,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:18,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:18,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:18,864 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:18,864 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:18,864 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:18,865 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,865 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:18,865 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:18,865 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:18,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:18,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:18,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:18,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:18,868 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:18,868 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:18,868 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:18,868 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:18,868 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:18,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:18,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:18,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:18,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:18,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:18,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:18,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:18,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:18,871 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:18,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:18,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:18,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:18,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:18,872 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:18,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:18,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,874 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,874 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:18,874 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:18,874 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:18,874 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:18,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:18,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:18,874 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,875 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:18,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:18,874 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:18,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:18,875 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:18,875 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:18,875 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:18,875 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:18,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:18,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:18,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:18,975 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:18,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:18,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:18,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:18,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:18,978 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:18,978 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:18,979 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:18,979 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:18,979 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:18,979 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:18,980 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:18,980 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:18,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:18,980 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:18,980 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:18,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:18,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:18,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:18,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:18,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:18,981 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:18,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:18,981 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:18,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:18,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:18,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:18,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:18,982 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:18,982 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:18,982 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e0314b11a1068d19a1052690701d1053 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:55:18,982 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=A 2023-04-27 03:55:18,982 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:18,982 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=B 2023-04-27 03:55:18,982 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:18,982 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=C 2023-04-27 03:55:18,982 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:18,996 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304278d45a4563de149e79208b9cfa4de6aaf to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304278d45a4563de149e79208b9cfa4de6aaf 2023-04-27 03:55:19,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:19,215 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:19,262 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:19,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567779251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567719236, max timestamp=1682567719261Num uniques 859; 2023-04-27 03:55:19,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567719236, max timestamp=1682567719261Num uniques 859; 2023-04-27 03:55:19,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567719236, max timestamp=1682567719261Num uniques 859; 2023-04-27 03:55:19,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:55:19,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:55:19,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:55:19,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:19,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567779309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:19,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567779310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:19,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567779316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:19,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567779364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,401 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=332, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/e1cb2b2db33b4bff8b78e538ac0454c2 2023-04-27 03:55:19,411 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/d289b402f3e34c3f9f86ab4ec86e7201 2023-04-27 03:55:19,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:19,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567779422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:19,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567779423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:19,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567779424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:19,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567779575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:19,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567779631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:19,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567779631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:19,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567779631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,819 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/ced9356217e54abe99f1e34c31cd9cf5 2023-04-27 03:55:19,823 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/e1cb2b2db33b4bff8b78e538ac0454c2 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/e1cb2b2db33b4bff8b78e538ac0454c2 2023-04-27 03:55:19,826 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/e1cb2b2db33b4bff8b78e538ac0454c2, entries=150, sequenceid=332, filesize=25.4 K 2023-04-27 03:55:19,827 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/d289b402f3e34c3f9f86ab4ec86e7201 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/d289b402f3e34c3f9f86ab4ec86e7201 2023-04-27 03:55:19,830 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/d289b402f3e34c3f9f86ab4ec86e7201, entries=150, sequenceid=332, filesize=11.9 K 2023-04-27 03:55:19,831 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/ced9356217e54abe99f1e34c31cd9cf5 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/ced9356217e54abe99f1e34c31cd9cf5 2023-04-27 03:55:19,833 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/ced9356217e54abe99f1e34c31cd9cf5, entries=150, sequenceid=332, filesize=11.9 K 2023-04-27 03:55:19,834 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e0314b11a1068d19a1052690701d1053 in 852ms, sequenceid=332, compaction requested=true 2023-04-27 03:55:19,834 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:19,834 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:19,834 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:19,834 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:19,834 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,834 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:19,834 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:19,837 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,837 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:19,837 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:19,837 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:19,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:19,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@399caa1a[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:19,838 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:19,838 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:19,839 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:19,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:19,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:19,840 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:19,840 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:19,840 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:19,840 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,840 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:19,841 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,841 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:19,841 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:19,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:19,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:19,841 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:19,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:19,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:19,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:19,843 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:19,843 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:19,844 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:19,844 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:19,844 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:19,845 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:19,845 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:19,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:19,845 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:19,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:19,846 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:19,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,847 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:19,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:19,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,849 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:19,849 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:19,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:19,850 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:19,850 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:19,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:19,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:19,850 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:19,850 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:19,850 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,850 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:19,850 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:19,850 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:19,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:19,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:19,850 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:19,850 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:19,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:19,893 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e0314b11a1068d19a1052690701d1053 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-27 03:55:19,893 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=A 2023-04-27 03:55:19,893 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:19,894 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=B 2023-04-27 03:55:19,894 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:19,894 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=C 2023-04-27 03:55:19,894 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:19,937 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427536adcfa85db4b489c3a90042b4c6543 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427536adcfa85db4b489c3a90042b4c6543 2023-04-27 03:55:19,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:19,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567779937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,950 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:19,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:19,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:19,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:19,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:19,954 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:19,954 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:19,954 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:19,954 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:19,954 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:19,955 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:19,956 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,956 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:19,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:19,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:19,956 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,956 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:19,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:19,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:19,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:19,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567779947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:19,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:19,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:19,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:19,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:19,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567779949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,964 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:19,964 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:19,964 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:19,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:19,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567779949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,964 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:19,964 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:19,964 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:19,964 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:19,964 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:19,964 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:19,965 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:19,965 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:19,965 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,965 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:19,965 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:19,966 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,966 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:19,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:19,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:19,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:19,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:19,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:19,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:19,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:19,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4737ae5e[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:19,968 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:19,968 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:19,969 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:19,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:19,969 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:19,969 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:19,969 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:19,969 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,970 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:19,972 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:19,972 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:19,972 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,972 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:19,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:19,977 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:19,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:19,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:19,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:19,977 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:19,980 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:19,980 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:19,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:19,980 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:19,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:19,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:19,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:19,981 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:19,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:19,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:19,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:19,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,982 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:19,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:19,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,984 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:19,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:19,984 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:19,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:19,984 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,984 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:19,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:19,985 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:19,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:19,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:19,985 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:19,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:19,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:19,985 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:19,985 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:19,985 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:19,985 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:19,985 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,062 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:20,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567780058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:20,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567780065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:20,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59768 deadline: 1682567780065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,085 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:20,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:20,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:20,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:20,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:20,088 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:20,088 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:20,088 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:20,088 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:20,089 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,089 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,090 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:20,090 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:20,090 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,090 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,091 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:20,091 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:20,091 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:20,091 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:20,091 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:20,091 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:20,092 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:20,092 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:20,092 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:20,092 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:20,092 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:20,092 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:20,092 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:20,092 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:20,092 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,092 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:20,092 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:20,095 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:20,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,095 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:20,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:20,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@217e5b89[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:20,096 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:20,097 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,098 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,098 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:20,098 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:20,098 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,098 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:20,098 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:20,099 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,099 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:20,099 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:20,099 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:20,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:20,101 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:20,101 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:20,101 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:20,101 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:20,101 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:20,103 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,103 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:20,103 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:20,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:20,104 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,105 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:20,110 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:20,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:20,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:20,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:20,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:20,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:20,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:20,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,110 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:20,111 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:20,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,111 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,111 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,211 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:20,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:20,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:20,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:20,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:20,214 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:20,214 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:20,214 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:20,214 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:20,214 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,214 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,215 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:20,215 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:20,216 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,216 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,216 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:20,217 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:20,217 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:20,217 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:20,217 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:20,217 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:20,217 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:20,217 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:20,217 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:20,217 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:20,217 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:20,217 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:20,217 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:20,217 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:20,217 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,217 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:20,217 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:20,219 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,219 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:20,219 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,219 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:20,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:20,220 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:20,220 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69a7c087[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:20,221 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,222 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:20,222 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:20,222 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:20,223 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,223 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:20,225 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:20,225 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,225 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:20,225 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:20,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:20,228 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:20,228 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:20,228 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:20,228 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:20,228 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:20,229 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,229 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:20,229 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:20,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:20,230 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:20,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:20,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:20,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,231 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:20,233 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:20,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:20,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:20,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:20,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:20,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,233 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:20,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:20,233 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:20,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:20,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:20,233 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,234 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,234 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,234 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,234 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:20,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567780263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:20,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567780272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:20,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59768 deadline: 1682567780273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,334 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:20,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:20,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:20,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:20,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:20,337 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:20,337 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:20,337 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:20,337 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:20,337 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,337 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,338 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,338 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:20,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:20,339 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,339 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:20,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:20,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:20,340 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:20,340 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:20,340 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:20,340 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:20,340 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:20,340 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:20,340 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:20,340 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:20,340 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:20,340 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:20,340 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:20,340 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,340 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:20,340 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:20,343 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,343 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:20,343 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,343 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:20,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:20,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35377bd3[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:20,344 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:20,344 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,346 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,346 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:20,346 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:20,346 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:20,347 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,347 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:20,348 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,348 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:20,348 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:20,348 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:20,348 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=369, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/0ba487a10ccb4abdabc5805d404a5add 2023-04-27 03:55:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:20,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:20,351 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:20,351 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:20,351 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:20,351 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:20,351 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:20,353 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:20,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:20,353 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,354 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:20,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:20,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:20,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:20,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,355 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:20,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,362 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,362 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,362 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,362 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:20,362 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:20,362 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:20,362 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:20,362 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:20,362 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,362 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:20,362 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:20,362 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:20,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:20,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:20,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567780454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,462 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:20,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:20,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:20,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:20,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:20,465 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:20,465 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:20,466 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:20,466 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:20,466 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,466 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,467 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,467 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:20,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,468 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,468 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:20,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,468 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:20,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:20,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:20,469 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:20,469 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:20,469 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:20,469 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:20,469 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:20,469 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:20,469 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:20,469 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:20,469 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:20,469 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:20,469 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:20,469 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,469 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:20,469 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:20,471 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,471 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:20,471 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,471 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:20,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:20,473 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:20,473 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f086675[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:20,474 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,474 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:20,474 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:20,474 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,474 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:20,474 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:20,476 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:20,476 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:20,476 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,476 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:20,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:20,478 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:20,478 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:20,478 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:20,478 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:20,478 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:20,480 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,480 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:20,480 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:20,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:20,481 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,482 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:20,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:20,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:20,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:20,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:20,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:20,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:20,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,484 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:20,484 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:20,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:20,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,484 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:20,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:20,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:20,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567780572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,584 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:20,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:20,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:20,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59768 deadline: 1682567780581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:20,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567780581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:20,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:20,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:20,588 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:20,588 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:20,588 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:20,588 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:20,589 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,589 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,593 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,593 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:20,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:20,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:20,593 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,593 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:20,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:20,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:20,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:20,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:20,594 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:20,594 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:20,594 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:20,595 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:20,595 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:20,595 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:20,595 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:20,595 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:20,595 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:20,595 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:20,595 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:20,595 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,595 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:20,595 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:20,597 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:20,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:20,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:20,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:20,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:20,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:20,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:20,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c79de0e[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:20,599 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:20,599 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,600 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,600 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:20,600 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,600 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:20,600 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:20,600 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:20,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:20,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:20,602 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:20,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:20,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:20,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:20,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:20,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:20,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:20,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:20,605 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:20,605 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:20,606 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:20,606 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:20,606 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:20,608 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,608 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:20,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,608 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:20,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,609 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,611 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:20,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:20,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:20,613 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:20,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:20,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:20,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,613 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:20,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:20,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:20,613 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:20,714 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:20,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:20,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:20,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:20,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:20,718 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:20,718 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:20,718 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:20,718 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:20,718 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,719 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,720 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:20,720 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,720 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,720 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:20,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:20,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:20,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:20,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:20,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:20,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:20,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:20,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:20,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:20,722 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:20,722 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:20,722 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:20,722 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:20,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:20,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:20,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:20,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:20,724 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,724 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:20,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:20,724 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,724 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:20,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:20,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:20,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:20,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:20,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b8d96fe[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:20,726 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:20,726 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,729 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:20,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:20,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:20,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:20,731 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:20,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:20,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:20,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:20,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:20,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:20,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:20,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:20,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:20,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:20,734 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:20,734 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:20,734 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:20,734 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:20,734 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:20,736 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,736 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:20,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,736 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:20,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:20,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:20,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:20,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:20,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:20,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,738 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:20,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,740 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:20,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:20,740 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:20,741 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:20,740 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,741 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:20,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:20,741 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:20,741 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:20,741 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:20,741 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,741 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,741 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,741 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:20,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:20,761 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=369 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/8bfa54fcaabd4bfeb7007dc9ec5e4412 2023-04-27 03:55:20,841 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:20,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:20,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:20,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:20,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:20,844 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:20,844 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:20,844 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:20,844 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:20,844 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,845 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,847 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,847 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:20,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:20,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:20,847 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,847 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:20,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:20,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:20,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:20,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:20,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:20,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:20,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:20,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:20,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:20,848 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:20,849 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:20,849 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:20,849 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:20,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:20,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:20,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:20,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:20,852 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,852 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:20,852 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,852 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:20,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:20,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45ea85bf[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:20,853 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:20,853 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,855 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:20,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:20,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:20,855 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,855 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:20,856 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:20,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:20,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:20,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:20,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:20,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:20,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:20,859 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:20,859 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:20,859 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:20,859 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:20,859 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:20,861 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,861 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:20,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,861 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:20,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:20,862 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,863 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:20,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,865 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,865 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,865 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,865 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:20,865 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:20,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:20,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:20,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:20,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:20,865 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:20,865 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:20,865 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,866 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:20,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:20,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:20,866 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:20,866 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,866 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,966 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:20,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:20,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:20,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:20,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:20,975 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:20,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:20,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:20,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:20,976 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,976 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,977 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:20,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:20,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:20,978 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,978 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:20,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:20,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:20,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:20,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:20,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:20,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:20,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:20,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:20,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:20,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:20,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:20,980 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:20,980 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:20,980 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:20,980 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:20,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:20,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:20,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:20,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:20,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,982 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:20,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:20,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:20,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:20,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:20,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:20,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:20,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49dd78b9[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:20,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,997 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:20,997 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:20,997 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:20,997 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:20,998 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,998 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:20,999 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,999 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:20,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:20,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:20,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:20,999 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:20,999 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:21,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:21,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:21,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:21,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:21,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:21,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:21,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:21,002 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:21,002 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:21,002 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:21,002 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:21,002 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:21,004 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:21,004 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:21,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:21,004 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:21,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:21,004 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:21,006 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:21,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,008 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:21,008 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:21,008 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,008 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:21,008 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:21,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:21,008 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:21,008 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,008 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:21,008 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:21,008 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:21,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:21,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:21,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:21,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:21,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:21,009 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:21,009 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:21,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:21,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567781083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:21,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567781089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:21,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59768 deadline: 1682567781090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,109 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:21,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:21,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:21,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:21,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:21,112 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:21,112 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:21,112 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:21,112 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:21,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:21,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:21,115 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,115 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:21,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:21,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:21,115 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,115 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:21,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:21,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:21,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:21,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:21,116 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:21,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:21,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:21,116 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:21,116 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:21,116 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:21,116 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:21,116 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:21,116 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:21,116 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:21,117 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:21,117 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:21,117 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:21,117 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:21,117 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,117 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:21,117 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:21,118 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:21,118 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:21,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:21,118 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:21,118 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:21,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:21,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:21,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:21,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:21,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:21,120 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:21,120 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:21,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b065622[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:21,121 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:21,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:21,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:21,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:21,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:21,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:21,122 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,122 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:21,123 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,123 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:21,123 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:21,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:21,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:21,123 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:21,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:21,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:21,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:21,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:21,125 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:21,126 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:21,126 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:21,126 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:21,126 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:21,128 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:21,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:21,129 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:21,129 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:21,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:21,130 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:21,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:21,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:21,131 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,134 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,134 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:21,134 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:21,134 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,134 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:21,134 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:21,134 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:21,134 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:21,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:21,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:21,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:21,134 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:21,134 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:21,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:21,135 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:21,135 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:21,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:21,172 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=369 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/7768ee29b5404c4ea71e0bfa2f6b6750 2023-04-27 03:55:21,175 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/0ba487a10ccb4abdabc5805d404a5add as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/0ba487a10ccb4abdabc5805d404a5add 2023-04-27 03:55:21,178 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/0ba487a10ccb4abdabc5805d404a5add, entries=200, sequenceid=369, filesize=32.3 K 2023-04-27 03:55:21,179 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/8bfa54fcaabd4bfeb7007dc9ec5e4412 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/8bfa54fcaabd4bfeb7007dc9ec5e4412 2023-04-27 03:55:21,186 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/8bfa54fcaabd4bfeb7007dc9ec5e4412, entries=150, sequenceid=369, filesize=11.9 K 2023-04-27 03:55:21,186 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/7768ee29b5404c4ea71e0bfa2f6b6750 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/7768ee29b5404c4ea71e0bfa2f6b6750 2023-04-27 03:55:21,196 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/7768ee29b5404c4ea71e0bfa2f6b6750, entries=150, sequenceid=369, filesize=11.9 K 2023-04-27 03:55:21,196 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=46.96 KB/48090 for e0314b11a1068d19a1052690701d1053 in 1303ms, sequenceid=369, compaction requested=true 2023-04-27 03:55:21,196 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:21,197 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:21,197 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:21,197 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-27 03:55:21,197 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-27 03:55:21,197 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:21,205 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 163596 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-27 03:55:21,205 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): e0314b11a1068d19a1052690701d1053/A is initiating minor compaction (all files) 2023-04-27 03:55:21,205 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e0314b11a1068d19a1052690701d1053/A in TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:21,205 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/f0ea729d72124b63b843973bdffb57f7, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/8a87a9b970be41c29990779fc018ff7b, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/929f831c612a48b9b3a38b1a6bf21221, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/723578ff32b0494c9e92aa0bc7d42344, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/e1cb2b2db33b4bff8b78e538ac0454c2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/0ba487a10ccb4abdabc5805d404a5add] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp, totalSize=159.8 K 2023-04-27 03:55:21,206 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73396 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-27 03:55:21,206 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e0314b11a1068d19a1052690701d1053/B is initiating minor compaction (all files) 2023-04-27 03:55:21,206 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e0314b11a1068d19a1052690701d1053/B in TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:21,206 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/5c484af2a41846ab84b1d29789c81fda, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/7bde2bb886a1473baad9e39e919135e0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/13e87b1b6fa74a9f909ae442dd7d765f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/6022e518344e48f7b5611b6700e12e99, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/d289b402f3e34c3f9f86ab4ec86e7201, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/8bfa54fcaabd4bfeb7007dc9ec5e4412] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp, totalSize=71.7 K 2023-04-27 03:55:21,206 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting f0ea729d72124b63b843973bdffb57f7, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1682567713777 2023-04-27 03:55:21,206 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 8a87a9b970be41c29990779fc018ff7b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1682567714048 2023-04-27 03:55:21,207 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 5c484af2a41846ab84b1d29789c81fda, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1682567713777 2023-04-27 03:55:21,207 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 929f831c612a48b9b3a38b1a6bf21221, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1682567716225 2023-04-27 03:55:21,207 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 7bde2bb886a1473baad9e39e919135e0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1682567714048 2023-04-27 03:55:21,208 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 723578ff32b0494c9e92aa0bc7d42344, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682567717437 2023-04-27 03:55:21,208 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 13e87b1b6fa74a9f909ae442dd7d765f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1682567716225 2023-04-27 03:55:21,208 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting e1cb2b2db33b4bff8b78e538ac0454c2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1682567718081 2023-04-27 03:55:21,208 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 6022e518344e48f7b5611b6700e12e99, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682567717437 2023-04-27 03:55:21,209 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 0ba487a10ccb4abdabc5805d404a5add, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1682567719295 2023-04-27 03:55:21,209 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting d289b402f3e34c3f9f86ab4ec86e7201, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1682567718081 2023-04-27 03:55:21,209 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 8bfa54fcaabd4bfeb7007dc9ec5e4412, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1682567719295 2023-04-27 03:55:21,233 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e0314b11a1068d19a1052690701d1053#B#compaction#376 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:21,235 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:21,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:21,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:21,238 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): e0314b11a1068d19a1052690701d1053#A#compaction#375 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:21,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:21,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:21,239 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:21,239 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:21,239 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:21,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:21,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:21,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:21,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,242 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,242 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:21,244 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:21,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:21,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:21,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:21,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:21,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:21,246 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:21,246 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:21,246 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:21,246 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:21,252 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:21,252 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:21,252 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:21,252 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:21,252 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e0314b11a1068d19a1052690701d1053 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:55:21,252 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=A 2023-04-27 03:55:21,252 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:21,252 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=B 2023-04-27 03:55:21,252 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:21,253 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=C 2023-04-27 03:55:21,253 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:21,274 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427a65bc443f06347e8848679e04416766a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427a65bc443f06347e8848679e04416766a 2023-04-27 03:55:21,285 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=379, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/1876808905c1450290180b4926774c03 2023-04-27 03:55:21,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:21,467 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:21,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567721523, max timestamp=1682567721613Num uniques 859; 2023-04-27 03:55:21,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567721523, max timestamp=1682567721613Num uniques 859; 2023-04-27 03:55:21,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567721523, max timestamp=1682567721613Num uniques 859; 2023-04-27 03:55:21,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:55:21,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:55:21,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:55:21,649 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/ff2abd5217b947b7be5704e19b165cc2 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/ff2abd5217b947b7be5704e19b165cc2 2023-04-27 03:55:21,658 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e0314b11a1068d19a1052690701d1053/B of e0314b11a1068d19a1052690701d1053 into ff2abd5217b947b7be5704e19b165cc2(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:21,659 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:21,659 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., storeName=e0314b11a1068d19a1052690701d1053/B, priority=10, startTime=1682567721197; duration=0sec 2023-04-27 03:55:21,659 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:21,659 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-27 03:55:21,662 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73396 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-27 03:55:21,662 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e0314b11a1068d19a1052690701d1053/C is initiating minor compaction (all files) 2023-04-27 03:55:21,662 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e0314b11a1068d19a1052690701d1053/C in TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:21,662 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/9393dfd093c2403bb8551e16913defb4, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/7dfe0d7c9b7e4af68463de29784ef5fa, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/eee4ec52c36e42bbbfd777beb7db0c77, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/c8ba9c9a5cd14a0cb0b3f42dfa5b94b8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/ced9356217e54abe99f1e34c31cd9cf5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/7768ee29b5404c4ea71e0bfa2f6b6750] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp, totalSize=71.7 K 2023-04-27 03:55:21,662 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 9393dfd093c2403bb8551e16913defb4, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1682567713777 2023-04-27 03:55:21,663 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 7dfe0d7c9b7e4af68463de29784ef5fa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1682567714048 2023-04-27 03:55:21,663 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting eee4ec52c36e42bbbfd777beb7db0c77, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1682567716225 2023-04-27 03:55:21,663 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting c8ba9c9a5cd14a0cb0b3f42dfa5b94b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682567717437 2023-04-27 03:55:21,664 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting ced9356217e54abe99f1e34c31cd9cf5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1682567718081 2023-04-27 03:55:21,664 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 7768ee29b5404c4ea71e0bfa2f6b6750, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1682567719295 2023-04-27 03:55:21,673 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e0314b11a1068d19a1052690701d1053#C#compaction#379 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:21,687 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/127d372d665e4ddf99bd7f349d8ae504 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/127d372d665e4ddf99bd7f349d8ae504 2023-04-27 03:55:21,691 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e0314b11a1068d19a1052690701d1053/C of e0314b11a1068d19a1052690701d1053 into 127d372d665e4ddf99bd7f349d8ae504(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:21,691 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:21,691 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., storeName=e0314b11a1068d19a1052690701d1053/C, priority=10, startTime=1682567721197; duration=0sec 2023-04-27 03:55:21,691 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:21,694 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/69ed13026fab4a2f9b1cec4f4051091d 2023-04-27 03:55:21,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:21,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567781791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:21,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:21,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567781896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:22,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:22,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567782093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:22,105 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:22,105 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/d713bc0074234d65888e6432c3b7c9dd 2023-04-27 03:55:22,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567782102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:22,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:22,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567782103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:22,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:22,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59768 deadline: 1682567782104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:22,109 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/1876808905c1450290180b4926774c03 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/1876808905c1450290180b4926774c03 2023-04-27 03:55:22,114 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/1876808905c1450290180b4926774c03, entries=150, sequenceid=379, filesize=25.4 K 2023-04-27 03:55:22,114 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/69ed13026fab4a2f9b1cec4f4051091d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/69ed13026fab4a2f9b1cec4f4051091d 2023-04-27 03:55:22,117 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/69ed13026fab4a2f9b1cec4f4051091d, entries=150, sequenceid=379, filesize=11.9 K 2023-04-27 03:55:22,118 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/d713bc0074234d65888e6432c3b7c9dd as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/d713bc0074234d65888e6432c3b7c9dd 2023-04-27 03:55:22,121 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/d713bc0074234d65888e6432c3b7c9dd, entries=150, sequenceid=379, filesize=11.9 K 2023-04-27 03:55:22,122 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e0314b11a1068d19a1052690701d1053 in 870ms, sequenceid=379, compaction requested=false 2023-04-27 03:55:22,122 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:22,122 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:22,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:22,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:22,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:22,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:22,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:22,123 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:22,123 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:22,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:22,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:22,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:22,124 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:22,124 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:22,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:22,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:22,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:22,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:22,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:22,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:22,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c9b09da[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:22,125 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:22,125 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:22,126 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:22,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:22,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:22,126 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:22,127 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:22,126 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:22,127 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:22,127 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:22,128 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:22,128 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:22,128 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:22,128 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:22,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:22,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:22,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:22,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:22,130 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:22,130 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:22,130 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:22,130 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:22,130 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:22,133 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:22,133 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:22,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:22,133 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:22,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:22,133 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:22,135 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:22,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:22,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:22,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:22,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:22,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:22,137 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:22,137 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:22,137 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:22,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:22,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:22,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:22,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:22,137 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:22,138 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:22,138 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:22,138 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:22,237 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:22,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:22,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:22,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:22,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:22,240 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:22,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:22,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:22,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:22,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:22,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:22,242 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:22,242 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:22,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:22,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:22,243 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:22,243 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:22,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:22,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:22,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:22,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:22,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:22,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:22,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:22,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:22,244 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:22,244 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:22,244 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:22,244 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e0314b11a1068d19a1052690701d1053 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-27 03:55:22,244 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=A 2023-04-27 03:55:22,244 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:22,244 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=B 2023-04-27 03:55:22,244 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:22,244 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=C 2023-04-27 03:55:22,244 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:22,266 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427b86597710b9a4d8e8ac13647bee6a5f7 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427b86597710b9a4d8e8ac13647bee6a5f7 2023-04-27 03:55:22,271 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=418, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/dbbaf3e52be0465b995ac9cae8f55b91 2023-04-27 03:55:22,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:22,414 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:22,461 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:22,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567782454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:22,465 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/3b13af6f630d4dbd9581f3ee6ce01ee0 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/3b13af6f630d4dbd9581f3ee6ce01ee0 2023-04-27 03:55:22,469 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e0314b11a1068d19a1052690701d1053/A of e0314b11a1068d19a1052690701d1053 into 3b13af6f630d4dbd9581f3ee6ce01ee0(size=26.2 K), total size for store is 51.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-27 03:55:22,469 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:22,469 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., storeName=e0314b11a1068d19a1052690701d1053/A, priority=10, startTime=1682567721197; duration=1sec 2023-04-27 03:55:22,469 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:22,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:22,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567782563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:22,680 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/3623cbbf3923450a84cd05ffe354b34e 2023-04-27 03:55:22,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:22,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567782768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:23,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:23,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567783073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:23,090 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/459fa33d82e04fbfb37ac53ef30e501a 2023-04-27 03:55:23,096 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/dbbaf3e52be0465b995ac9cae8f55b91 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/dbbaf3e52be0465b995ac9cae8f55b91 2023-04-27 03:55:23,100 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/dbbaf3e52be0465b995ac9cae8f55b91, entries=150, sequenceid=418, filesize=25.4 K 2023-04-27 03:55:23,101 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/3623cbbf3923450a84cd05ffe354b34e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/3623cbbf3923450a84cd05ffe354b34e 2023-04-27 03:55:23,110 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/3623cbbf3923450a84cd05ffe354b34e, entries=150, sequenceid=418, filesize=11.9 K 2023-04-27 03:55:23,111 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/459fa33d82e04fbfb37ac53ef30e501a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/459fa33d82e04fbfb37ac53ef30e501a 2023-04-27 03:55:23,117 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/459fa33d82e04fbfb37ac53ef30e501a, entries=150, sequenceid=418, filesize=11.9 K 2023-04-27 03:55:23,118 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for e0314b11a1068d19a1052690701d1053 in 874ms, sequenceid=418, compaction requested=true 2023-04-27 03:55:23,118 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:23,118 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:23,118 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:23,118 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:23,118 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:23,118 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:23,118 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:23,120 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:23,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:23,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:23,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:23,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:23,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d4da48b[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:23,127 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:23,127 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:23,133 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:23,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:23,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:23,133 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:23,133 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:23,133 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:23,133 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:23,133 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:23,142 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:23,143 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:23,142 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:23,143 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:23,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:23,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:23,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:23,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:23,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:23,147 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:23,147 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:23,147 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:23,147 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:23,148 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:23,149 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:23,149 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:23,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:23,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:23,150 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:23,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:23,152 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:23,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:23,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:23,154 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:23,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:23,157 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:23,157 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:23,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:23,157 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:23,157 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:23,157 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:23,157 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:23,157 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:23,157 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:23,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:23,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:23,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:23,157 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:23,157 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:23,157 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:23,158 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:23,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:23,258 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:23,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:23,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:23,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:23,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:23,261 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:23,261 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:23,261 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:23,261 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:23,261 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:23,261 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:23,265 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:23,265 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:23,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:23,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:23,265 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:23,266 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:23,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:23,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:23,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:23,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:23,266 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:23,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:23,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:23,270 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:23,270 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:23,270 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:23,271 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:23,271 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:23,271 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:23,271 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:23,271 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e0314b11a1068d19a1052690701d1053 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:55:23,272 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=A 2023-04-27 03:55:23,272 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:23,272 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=B 2023-04-27 03:55:23,272 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:23,272 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=C 2023-04-27 03:55:23,272 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:23,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:23,582 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:23,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567723642, max timestamp=1682567723669Num uniques 859; 2023-04-27 03:55:23,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567723642, max timestamp=1682567723669Num uniques 859; 2023-04-27 03:55:23,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:55:23,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:55:23,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567723642, max timestamp=1682567723669Num uniques 859; 2023-04-27 03:55:23,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:55:23,687 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304272d1e1a62adb14a85b5242e349ac8dba9 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272d1e1a62adb14a85b5242e349ac8dba9 2023-04-27 03:55:23,692 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=429, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/1626063254394a2db772843b16f800ac 2023-04-27 03:55:23,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:23,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567783797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:23,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:23,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567783905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:24,101 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/3af63fd9e63340fb89fad62db41df5ee 2023-04-27 03:55:24,117 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:24,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59798 deadline: 1682567784110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:24,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:24,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567784111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:24,120 DEBUG [Thread-2450] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4169 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., hostname=jenkins-hbase4.apache.org,40885,1682567618305, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 03:55:24,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:24,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59784 deadline: 1682567784115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:24,133 DEBUG [Thread-2458] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4183 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., hostname=jenkins-hbase4.apache.org,40885,1682567618305, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 03:55:24,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:24,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59768 deadline: 1682567784127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:24,141 DEBUG [Thread-2452] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4190 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., hostname=jenkins-hbase4.apache.org,40885,1682567618305, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 03:55:24,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:24,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567784422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:24,518 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/5e3d80b2f490430e977514e5aef153be 2023-04-27 03:55:24,521 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/1626063254394a2db772843b16f800ac as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/1626063254394a2db772843b16f800ac 2023-04-27 03:55:24,524 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/1626063254394a2db772843b16f800ac, entries=150, sequenceid=429, filesize=25.4 K 2023-04-27 03:55:24,525 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/3af63fd9e63340fb89fad62db41df5ee as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/3af63fd9e63340fb89fad62db41df5ee 2023-04-27 03:55:24,529 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/3af63fd9e63340fb89fad62db41df5ee, entries=150, sequenceid=429, filesize=11.9 K 2023-04-27 03:55:24,530 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/5e3d80b2f490430e977514e5aef153be as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/5e3d80b2f490430e977514e5aef153be 2023-04-27 03:55:24,533 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/5e3d80b2f490430e977514e5aef153be, entries=150, sequenceid=429, filesize=11.9 K 2023-04-27 03:55:24,533 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e0314b11a1068d19a1052690701d1053 in 1262ms, sequenceid=429, compaction requested=true 2023-04-27 03:55:24,534 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:24,534 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:24,534 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:24,534 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:24,534 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:24,534 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:24,534 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:24,535 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:24,535 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:24,536 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:24,536 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:24,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:24,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:24,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:24,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ff7192b[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:24,537 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:24,537 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:24,538 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:24,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:24,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:24,538 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:24,538 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:24,538 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:24,539 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:24,538 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:24,540 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:24,540 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:24,540 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:24,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:24,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:24,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:24,540 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:24,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:24,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:24,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:24,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:24,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:24,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:24,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:24,542 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:24,542 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:24,542 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:24,542 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:24,542 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:24,544 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:24,544 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:24,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:24,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:24,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:24,544 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:24,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:24,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:24,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:24,545 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:24,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:24,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:24,546 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:24,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:24,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:24,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:24,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:24,549 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:24,549 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:24,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:24,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:24,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:24,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:24,549 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:24,549 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:24,550 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:24,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:24,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:24,556 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:24,556 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:24,556 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:24,556 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:24,556 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:24,556 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:24,556 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:24,650 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:24,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:24,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:24,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:24,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:24,653 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:24,653 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:24,653 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:24,653 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:24,653 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:24,653 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:24,655 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:24,655 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:24,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:24,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:24,655 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:24,655 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:24,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:24,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:24,656 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:24,656 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:24,656 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:24,656 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:24,656 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:24,656 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:24,656 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:24,656 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:24,656 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:24,656 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. started... 2023-04-27 03:55:24,657 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e0314b11a1068d19a1052690701d1053 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-27 03:55:24,657 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=A 2023-04-27 03:55:24,657 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:24,657 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=B 2023-04-27 03:55:24,657 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:24,657 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=C 2023-04-27 03:55:24,657 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:24,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:24,936 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. as already flushing 2023-04-27 03:55:24,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:24,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567784997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:25,068 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304272eb01b5441ea4d958feb151449272fef to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272eb01b5441ea4d958feb151449272fef 2023-04-27 03:55:25,104 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:25,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567785099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:25,165 DEBUG [Thread-2465] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d7ba242 to 127.0.0.1:55020 2023-04-27 03:55:25,165 DEBUG [Thread-2465] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:55:25,170 DEBUG [Thread-2463] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ec12b02 to 127.0.0.1:55020 2023-04-27 03:55:25,170 DEBUG [Thread-2463] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:55:25,172 DEBUG [Thread-2467] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e079da1 to 127.0.0.1:55020 2023-04-27 03:55:25,172 DEBUG [Thread-2467] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:55:25,174 DEBUG [Thread-2469] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x159c5e0c to 127.0.0.1:55020 2023-04-27 03:55:25,174 DEBUG [Thread-2469] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:55:25,178 DEBUG [Thread-2461] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x68a0e21a to 127.0.0.1:55020 2023-04-27 03:55:25,178 DEBUG [Thread-2461] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:55:25,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:25,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567785306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:25,476 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=466, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/64872fb4beed4c459c022d1b34508ad2 2023-04-27 03:55:25,608 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:25,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567785608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:25,884 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/ca60034b4805405c8eb5298207afdc92 2023-04-27 03:55:26,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:26,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59742 deadline: 1682567786112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:26,292 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/24b8557fada04823ab7df4c2179877d9 2023-04-27 03:55:26,295 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/64872fb4beed4c459c022d1b34508ad2 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/64872fb4beed4c459c022d1b34508ad2 2023-04-27 03:55:26,298 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/64872fb4beed4c459c022d1b34508ad2, entries=150, sequenceid=466, filesize=25.4 K 2023-04-27 03:55:26,298 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/ca60034b4805405c8eb5298207afdc92 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/ca60034b4805405c8eb5298207afdc92 2023-04-27 03:55:26,301 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/ca60034b4805405c8eb5298207afdc92, entries=150, sequenceid=466, filesize=11.9 K 2023-04-27 03:55:26,301 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/24b8557fada04823ab7df4c2179877d9 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/24b8557fada04823ab7df4c2179877d9 2023-04-27 03:55:26,304 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/24b8557fada04823ab7df4c2179877d9, entries=150, sequenceid=466, filesize=11.9 K 2023-04-27 03:55:26,305 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for e0314b11a1068d19a1052690701d1053 in 1648ms, sequenceid=466, compaction requested=true 2023-04-27 03:55:26,305 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:26,305 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:26,307 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:26,307 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:26,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:26,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:26,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:26,307 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:26,307 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:26,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:26,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:26,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:26,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:26,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:26,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:26,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2669e9b1[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:26,308 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:26,309 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:26,309 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:26,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:26,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:26,310 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:26,310 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:26,310 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:26,310 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:26,310 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:26,311 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:26,311 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:26,311 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:26,311 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:26,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:26,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:26,313 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:26,313 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:26,313 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:26,313 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:26,313 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:26,314 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:26,315 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:26,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:26,315 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:26,315 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:26,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:26,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:26,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:26,316 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:26,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:26,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:26,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:26,319 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:26,319 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:26,319 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:26,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:26,319 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:26,319 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:26,319 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:26,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:26,319 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:26,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:26,319 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:26,319 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:26,319 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:26,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:26,319 DEBUG [Thread-2460] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:26,319 DEBUG [Thread-2460] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:26,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:26,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:26,321 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e0314b11a1068d19a1052690701d1053 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-27 03:55:26,321 DEBUG [Thread-2456] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fe1db6a to 127.0.0.1:55020 2023-04-27 03:55:26,321 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=A 2023-04-27 03:55:26,321 DEBUG [Thread-2456] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:55:26,321 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:26,321 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=B 2023-04-27 03:55:26,321 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:26,321 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=C 2023-04-27 03:55:26,321 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:26,419 DEBUG [Thread-2460] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:26,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:26,732 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427eb6fffdf6caa4a929fe28da11b276001 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427eb6fffdf6caa4a929fe28da11b276001 2023-04-27 03:55:27,116 DEBUG [Thread-2454] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3620ee35 to 127.0.0.1:55020 2023-04-27 03:55:27,116 DEBUG [Thread-2454] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:55:27,137 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=477, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/8b5a8abed1de4cd19bd3713e693fee90 2023-04-27 03:55:27,545 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/dfebd9c80ea44baab16e9ad9d53f5fab 2023-04-27 03:55:27,953 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/1f4b55ca232848d6a1246450a32d03fc 2023-04-27 03:55:27,957 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/8b5a8abed1de4cd19bd3713e693fee90 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/8b5a8abed1de4cd19bd3713e693fee90 2023-04-27 03:55:27,959 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/8b5a8abed1de4cd19bd3713e693fee90, entries=150, sequenceid=477, filesize=25.4 K 2023-04-27 03:55:27,960 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/dfebd9c80ea44baab16e9ad9d53f5fab as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/dfebd9c80ea44baab16e9ad9d53f5fab 2023-04-27 03:55:27,962 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/dfebd9c80ea44baab16e9ad9d53f5fab, entries=150, sequenceid=477, filesize=11.9 K 2023-04-27 03:55:27,963 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/1f4b55ca232848d6a1246450a32d03fc as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/1f4b55ca232848d6a1246450a32d03fc 2023-04-27 03:55:27,965 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/1f4b55ca232848d6a1246450a32d03fc, entries=150, sequenceid=477, filesize=11.9 K 2023-04-27 03:55:27,966 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=6.71 KB/6870 for e0314b11a1068d19a1052690701d1053 in 1645ms, sequenceid=477, compaction requested=true 2023-04-27 03:55:27,966 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:27,966 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:27,966 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:27,966 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-27 03:55:27,966 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-27 03:55:27,966 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:27,967 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 157060 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-27 03:55:27,968 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73900 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-27 03:55:27,968 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): e0314b11a1068d19a1052690701d1053/A is initiating minor compaction (all files) 2023-04-27 03:55:27,968 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): e0314b11a1068d19a1052690701d1053/B is initiating minor compaction (all files) 2023-04-27 03:55:27,968 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e0314b11a1068d19a1052690701d1053/A in TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:27,968 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e0314b11a1068d19a1052690701d1053/B in TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:27,968 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/3b13af6f630d4dbd9581f3ee6ce01ee0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/1876808905c1450290180b4926774c03, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/dbbaf3e52be0465b995ac9cae8f55b91, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/1626063254394a2db772843b16f800ac, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/64872fb4beed4c459c022d1b34508ad2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/8b5a8abed1de4cd19bd3713e693fee90] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp, totalSize=153.4 K 2023-04-27 03:55:27,968 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/ff2abd5217b947b7be5704e19b165cc2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/69ed13026fab4a2f9b1cec4f4051091d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/3623cbbf3923450a84cd05ffe354b34e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/3af63fd9e63340fb89fad62db41df5ee, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/ca60034b4805405c8eb5298207afdc92, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/dfebd9c80ea44baab16e9ad9d53f5fab] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp, totalSize=72.2 K 2023-04-27 03:55:27,968 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 3b13af6f630d4dbd9581f3ee6ce01ee0, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1682567719295 2023-04-27 03:55:27,968 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting ff2abd5217b947b7be5704e19b165cc2, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1682567719295 2023-04-27 03:55:27,968 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 1876808905c1450290180b4926774c03, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1682567719933 2023-04-27 03:55:27,968 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 69ed13026fab4a2f9b1cec4f4051091d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1682567719933 2023-04-27 03:55:27,968 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting dbbaf3e52be0465b995ac9cae8f55b91, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1682567721711 2023-04-27 03:55:27,969 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 3623cbbf3923450a84cd05ffe354b34e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1682567721711 2023-04-27 03:55:27,969 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 1626063254394a2db772843b16f800ac, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1682567722423 2023-04-27 03:55:27,969 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 3af63fd9e63340fb89fad62db41df5ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1682567722423 2023-04-27 03:55:27,969 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 64872fb4beed4c459c022d1b34508ad2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1682567723783 2023-04-27 03:55:27,969 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting ca60034b4805405c8eb5298207afdc92, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1682567723783 2023-04-27 03:55:27,969 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 8b5a8abed1de4cd19bd3713e693fee90, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1682567724988 2023-04-27 03:55:27,969 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting dfebd9c80ea44baab16e9ad9d53f5fab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1682567724988 2023-04-27 03:55:27,978 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): e0314b11a1068d19a1052690701d1053#B#compaction#394 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:27,990 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): e0314b11a1068d19a1052690701d1053#A#compaction#393 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:28,127 DEBUG [Thread-2450] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38341134 to 127.0.0.1:55020 2023-04-27 03:55:28,127 DEBUG [Thread-2450] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:55:28,145 DEBUG [Thread-2452] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x71b5058b to 127.0.0.1:55020 2023-04-27 03:55:28,145 DEBUG [Thread-2452] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:55:28,169 DEBUG [Thread-2458] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10377b50 to 127.0.0.1:55020 2023-04-27 03:55:28,169 DEBUG [Thread-2458] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:55:28,169 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-27 03:55:28,169 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 48 2023-04-27 03:55:28,169 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 61 2023-04-27 03:55:28,169 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 220 2023-04-27 03:55:28,169 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 20 2023-04-27 03:55:28,169 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 40 2023-04-27 03:55:28,169 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-27 03:55:28,169 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-27 03:55:28,169 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(402): scanned 1832 2023-04-27 03:55:28,170 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(403): verified 5496 rows 2023-04-27 03:55:28,170 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(402): scanned 1828 2023-04-27 03:55:28,170 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(403): verified 5484 rows 2023-04-27 03:55:28,170 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(402): scanned 1829 2023-04-27 03:55:28,170 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(403): verified 5487 rows 2023-04-27 03:55:28,170 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(402): scanned 1835 2023-04-27 03:55:28,170 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(403): verified 5505 rows 2023-04-27 03:55:28,170 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(402): scanned 1827 2023-04-27 03:55:28,170 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(403): verified 5481 rows 2023-04-27 03:55:28,170 INFO [Listener at localhost/33933] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-27 03:55:28,170 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bcfc1e0 to 127.0.0.1:55020 2023-04-27 03:55:28,170 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:55:28,172 INFO [Listener at localhost/33933] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-27 03:55:28,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-27 03:55:28,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-27 03:55:28,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-27 03:55:28,177 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567728177"}]},"ts":"1682567728177"} 2023-04-27 03:55:28,178 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-27 03:55:28,180 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-27 03:55:28,181 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e0314b11a1068d19a1052690701d1053, UNASSIGN}] 2023-04-27 03:55:28,182 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e0314b11a1068d19a1052690701d1053, UNASSIGN 2023-04-27 03:55:28,182 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=e0314b11a1068d19a1052690701d1053, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:28,183 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567728182"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682567728182"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682567728182"}]},"ts":"1682567728182"} 2023-04-27 03:55:28,183 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305}] 2023-04-27 03:55:28,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-27 03:55:28,335 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:28,335 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e0314b11a1068d19a1052690701d1053, disabling compactions & flushes 2023-04-27 03:55:28,335 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 2 compactions to complete for region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:28,394 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/1e12314615b44c6796f744f6947adc6e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/1e12314615b44c6796f744f6947adc6e 2023-04-27 03:55:28,398 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e0314b11a1068d19a1052690701d1053/B of e0314b11a1068d19a1052690701d1053 into 1e12314615b44c6796f744f6947adc6e(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:28,398 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:28,398 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., storeName=e0314b11a1068d19a1052690701d1053/B, priority=10, startTime=1682567727966; duration=0sec 2023-04-27 03:55:28,399 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:28,399 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. because compaction request was cancelled 2023-04-27 03:55:28,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-27 03:55:28,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-27 03:55:29,207 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/f515f9916124477dad4a2bc5e08ece8d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/f515f9916124477dad4a2bc5e08ece8d 2023-04-27 03:55:29,210 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e0314b11a1068d19a1052690701d1053/A of e0314b11a1068d19a1052690701d1053 into f515f9916124477dad4a2bc5e08ece8d(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-27 03:55:29,210 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:29,210 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:29,210 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053., storeName=e0314b11a1068d19a1052690701d1053/A, priority=10, startTime=1682567727966; duration=1sec 2023-04-27 03:55:29,210 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:29,210 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. after waiting 0 ms 2023-04-27 03:55:29,210 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:29,210 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:29,210 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing e0314b11a1068d19a1052690701d1053 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-27 03:55:29,210 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=A 2023-04-27 03:55:29,210 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:29,210 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=B 2023-04-27 03:55:29,210 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:29,210 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e0314b11a1068d19a1052690701d1053, store=C 2023-04-27 03:55:29,210 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:29,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-27 03:55:29,620 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427f26fee1ef3084545bc32162c2524e728 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427f26fee1ef3084545bc32162c2524e728 2023-04-27 03:55:30,025 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=486, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/b7846695d2144130bcb67e23a0bdeb91 2023-04-27 03:55:30,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-27 03:55:30,435 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/efa76fc2ea4d42c487598921cb638181 2023-04-27 03:55:30,843 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/5882d39adba74b019988488ba89e1ee9 2023-04-27 03:55:30,847 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/A/b7846695d2144130bcb67e23a0bdeb91 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/b7846695d2144130bcb67e23a0bdeb91 2023-04-27 03:55:30,851 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/b7846695d2144130bcb67e23a0bdeb91, entries=150, sequenceid=486, filesize=25.4 K 2023-04-27 03:55:30,851 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/B/efa76fc2ea4d42c487598921cb638181 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/efa76fc2ea4d42c487598921cb638181 2023-04-27 03:55:30,855 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/efa76fc2ea4d42c487598921cb638181, entries=150, sequenceid=486, filesize=11.9 K 2023-04-27 03:55:30,856 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/.tmp/C/5882d39adba74b019988488ba89e1ee9 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/5882d39adba74b019988488ba89e1ee9 2023-04-27 03:55:30,859 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/5882d39adba74b019988488ba89e1ee9, entries=150, sequenceid=486, filesize=11.9 K 2023-04-27 03:55:30,860 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for e0314b11a1068d19a1052690701d1053 in 1650ms, sequenceid=486, compaction requested=true 2023-04-27 03:55:30,888 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/1b46db9689f14125a42575828a9a5f69, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/285abf99c1244b518792bda18c4e7497, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/d2cf3fa316da4a65a9d2437f8fd8bd7e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/8196600520d94233a9b04e47216cd086, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/1c7c16c73f984b449bcbeac9a672465f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/d59e454f8a5543d1a799dda7e503fd88, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/36647eebafe2494fa84d245032b9e151, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/f24e1932c8fe4444857a58e3d33ef6da, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/ffdff68fae094f81a8b280055b3bd294, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/35f8386ff0fb48f3b20c27cd2aec6d44, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/b7a7f1d9c640483d8e059408d51f2b52, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/a93bdbf438fe472ca751f3e1d414fa38, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/c97bf34fdaaa4392a1a59d1aee81aecc, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/8e9c91e83bcc4e6396588160181bd390, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/2cb4e0250cf84600931b7c72471d3c50, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/11b002c77c7a4fc18ad536615b61ad68, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/bc9f6bd12a4e408a86a169eae71802bf, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/f0ea729d72124b63b843973bdffb57f7, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/8a87a9b970be41c29990779fc018ff7b, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/929f831c612a48b9b3a38b1a6bf21221, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/723578ff32b0494c9e92aa0bc7d42344, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/e1cb2b2db33b4bff8b78e538ac0454c2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/0ba487a10ccb4abdabc5805d404a5add, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/3b13af6f630d4dbd9581f3ee6ce01ee0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/1876808905c1450290180b4926774c03, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/dbbaf3e52be0465b995ac9cae8f55b91, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/1626063254394a2db772843b16f800ac, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/64872fb4beed4c459c022d1b34508ad2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/8b5a8abed1de4cd19bd3713e693fee90] to archive 2023-04-27 03:55:30,889 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 03:55:30,891 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/1b46db9689f14125a42575828a9a5f69 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/1b46db9689f14125a42575828a9a5f69 2023-04-27 03:55:30,892 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/285abf99c1244b518792bda18c4e7497 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/285abf99c1244b518792bda18c4e7497 2023-04-27 03:55:30,893 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/d2cf3fa316da4a65a9d2437f8fd8bd7e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/d2cf3fa316da4a65a9d2437f8fd8bd7e 2023-04-27 03:55:30,895 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/8196600520d94233a9b04e47216cd086 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/8196600520d94233a9b04e47216cd086 2023-04-27 03:55:30,896 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/1c7c16c73f984b449bcbeac9a672465f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/1c7c16c73f984b449bcbeac9a672465f 2023-04-27 03:55:30,897 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/d59e454f8a5543d1a799dda7e503fd88 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/d59e454f8a5543d1a799dda7e503fd88 2023-04-27 03:55:30,898 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/36647eebafe2494fa84d245032b9e151 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/36647eebafe2494fa84d245032b9e151 2023-04-27 03:55:30,899 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/f24e1932c8fe4444857a58e3d33ef6da to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/f24e1932c8fe4444857a58e3d33ef6da 2023-04-27 03:55:30,900 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/ffdff68fae094f81a8b280055b3bd294 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/ffdff68fae094f81a8b280055b3bd294 2023-04-27 03:55:30,900 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/35f8386ff0fb48f3b20c27cd2aec6d44 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/35f8386ff0fb48f3b20c27cd2aec6d44 2023-04-27 03:55:30,901 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/b7a7f1d9c640483d8e059408d51f2b52 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/b7a7f1d9c640483d8e059408d51f2b52 2023-04-27 03:55:30,902 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/a93bdbf438fe472ca751f3e1d414fa38 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/a93bdbf438fe472ca751f3e1d414fa38 2023-04-27 03:55:30,903 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/c97bf34fdaaa4392a1a59d1aee81aecc to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/c97bf34fdaaa4392a1a59d1aee81aecc 2023-04-27 03:55:30,904 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/8e9c91e83bcc4e6396588160181bd390 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/8e9c91e83bcc4e6396588160181bd390 2023-04-27 03:55:30,905 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/2cb4e0250cf84600931b7c72471d3c50 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/2cb4e0250cf84600931b7c72471d3c50 2023-04-27 03:55:30,906 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/11b002c77c7a4fc18ad536615b61ad68 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/11b002c77c7a4fc18ad536615b61ad68 2023-04-27 03:55:30,907 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/bc9f6bd12a4e408a86a169eae71802bf to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/bc9f6bd12a4e408a86a169eae71802bf 2023-04-27 03:55:30,908 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/f0ea729d72124b63b843973bdffb57f7 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/f0ea729d72124b63b843973bdffb57f7 2023-04-27 03:55:30,909 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/8a87a9b970be41c29990779fc018ff7b to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/8a87a9b970be41c29990779fc018ff7b 2023-04-27 03:55:30,910 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/929f831c612a48b9b3a38b1a6bf21221 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/929f831c612a48b9b3a38b1a6bf21221 2023-04-27 03:55:30,911 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/723578ff32b0494c9e92aa0bc7d42344 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/723578ff32b0494c9e92aa0bc7d42344 2023-04-27 03:55:30,912 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/e1cb2b2db33b4bff8b78e538ac0454c2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/e1cb2b2db33b4bff8b78e538ac0454c2 2023-04-27 03:55:30,913 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/0ba487a10ccb4abdabc5805d404a5add to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/0ba487a10ccb4abdabc5805d404a5add 2023-04-27 03:55:30,914 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/3b13af6f630d4dbd9581f3ee6ce01ee0 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/3b13af6f630d4dbd9581f3ee6ce01ee0 2023-04-27 03:55:30,915 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/1876808905c1450290180b4926774c03 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/1876808905c1450290180b4926774c03 2023-04-27 03:55:30,916 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/dbbaf3e52be0465b995ac9cae8f55b91 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/dbbaf3e52be0465b995ac9cae8f55b91 2023-04-27 03:55:30,917 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/1626063254394a2db772843b16f800ac to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/1626063254394a2db772843b16f800ac 2023-04-27 03:55:30,918 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/64872fb4beed4c459c022d1b34508ad2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/64872fb4beed4c459c022d1b34508ad2 2023-04-27 03:55:30,919 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/8b5a8abed1de4cd19bd3713e693fee90 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/8b5a8abed1de4cd19bd3713e693fee90 2023-04-27 03:55:30,921 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/5c19e4af3dc147498eab4edfc8f69114, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/d0c769c48a19435782ba3202fe68d5c4, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/f8aa684a3260406eb20279223e03caea, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/e381188d9ade43db8e0624b39e8bfccb, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/d0177ad1e0694d43afc2d784fae38424, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/4d9d7ac79ab141578a1d68aef018ec92, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/8e8d071a65f846adabbbd43e761ea8f7, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/76609995604646f299e5f56b46cf75ad, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/68301c30cbe1434597c3b2e4345eba3f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/e2ab8078edd64bdf89d9f51d9ff662d0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/98ec8be65f11438d896ecca506d0a6ac, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/5a461389fded4a16aa7fffb53f7bd96d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/5bd8c687fcbd42cbbc8098fc66db6f34, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/efc3a3fda3b742918bb8153d78279716, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/e9098b676caf4e54a74b379073818371, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/9f165d5d1e1145ed85db124fdbbe8aef, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/5c484af2a41846ab84b1d29789c81fda, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/36df2ac31305402995405d4bd23f948f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/7bde2bb886a1473baad9e39e919135e0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/13e87b1b6fa74a9f909ae442dd7d765f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/6022e518344e48f7b5611b6700e12e99, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/d289b402f3e34c3f9f86ab4ec86e7201, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/ff2abd5217b947b7be5704e19b165cc2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/8bfa54fcaabd4bfeb7007dc9ec5e4412, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/69ed13026fab4a2f9b1cec4f4051091d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/3623cbbf3923450a84cd05ffe354b34e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/3af63fd9e63340fb89fad62db41df5ee, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/ca60034b4805405c8eb5298207afdc92, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/dfebd9c80ea44baab16e9ad9d53f5fab] to archive 2023-04-27 03:55:30,922 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 03:55:30,923 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/5c19e4af3dc147498eab4edfc8f69114 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/5c19e4af3dc147498eab4edfc8f69114 2023-04-27 03:55:30,924 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/d0c769c48a19435782ba3202fe68d5c4 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/d0c769c48a19435782ba3202fe68d5c4 2023-04-27 03:55:30,925 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/f8aa684a3260406eb20279223e03caea to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/f8aa684a3260406eb20279223e03caea 2023-04-27 03:55:30,926 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/e381188d9ade43db8e0624b39e8bfccb to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/e381188d9ade43db8e0624b39e8bfccb 2023-04-27 03:55:30,926 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/d0177ad1e0694d43afc2d784fae38424 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/d0177ad1e0694d43afc2d784fae38424 2023-04-27 03:55:30,927 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/4d9d7ac79ab141578a1d68aef018ec92 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/4d9d7ac79ab141578a1d68aef018ec92 2023-04-27 03:55:30,928 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/8e8d071a65f846adabbbd43e761ea8f7 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/8e8d071a65f846adabbbd43e761ea8f7 2023-04-27 03:55:30,929 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/76609995604646f299e5f56b46cf75ad to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/76609995604646f299e5f56b46cf75ad 2023-04-27 03:55:30,930 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/68301c30cbe1434597c3b2e4345eba3f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/68301c30cbe1434597c3b2e4345eba3f 2023-04-27 03:55:30,931 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/e2ab8078edd64bdf89d9f51d9ff662d0 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/e2ab8078edd64bdf89d9f51d9ff662d0 2023-04-27 03:55:30,932 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/98ec8be65f11438d896ecca506d0a6ac to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/98ec8be65f11438d896ecca506d0a6ac 2023-04-27 03:55:30,932 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/5a461389fded4a16aa7fffb53f7bd96d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/5a461389fded4a16aa7fffb53f7bd96d 2023-04-27 03:55:30,933 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/5bd8c687fcbd42cbbc8098fc66db6f34 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/5bd8c687fcbd42cbbc8098fc66db6f34 2023-04-27 03:55:30,934 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/efc3a3fda3b742918bb8153d78279716 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/efc3a3fda3b742918bb8153d78279716 2023-04-27 03:55:30,935 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/e9098b676caf4e54a74b379073818371 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/e9098b676caf4e54a74b379073818371 2023-04-27 03:55:30,936 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/9f165d5d1e1145ed85db124fdbbe8aef to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/9f165d5d1e1145ed85db124fdbbe8aef 2023-04-27 03:55:30,936 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/5c484af2a41846ab84b1d29789c81fda to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/5c484af2a41846ab84b1d29789c81fda 2023-04-27 03:55:30,937 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/36df2ac31305402995405d4bd23f948f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/36df2ac31305402995405d4bd23f948f 2023-04-27 03:55:30,938 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/7bde2bb886a1473baad9e39e919135e0 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/7bde2bb886a1473baad9e39e919135e0 2023-04-27 03:55:30,938 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/13e87b1b6fa74a9f909ae442dd7d765f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/13e87b1b6fa74a9f909ae442dd7d765f 2023-04-27 03:55:30,939 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/6022e518344e48f7b5611b6700e12e99 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/6022e518344e48f7b5611b6700e12e99 2023-04-27 03:55:30,940 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/d289b402f3e34c3f9f86ab4ec86e7201 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/d289b402f3e34c3f9f86ab4ec86e7201 2023-04-27 03:55:30,941 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/ff2abd5217b947b7be5704e19b165cc2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/ff2abd5217b947b7be5704e19b165cc2 2023-04-27 03:55:30,942 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/8bfa54fcaabd4bfeb7007dc9ec5e4412 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/8bfa54fcaabd4bfeb7007dc9ec5e4412 2023-04-27 03:55:30,943 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/69ed13026fab4a2f9b1cec4f4051091d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/69ed13026fab4a2f9b1cec4f4051091d 2023-04-27 03:55:30,944 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/3623cbbf3923450a84cd05ffe354b34e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/3623cbbf3923450a84cd05ffe354b34e 2023-04-27 03:55:30,945 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/3af63fd9e63340fb89fad62db41df5ee to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/3af63fd9e63340fb89fad62db41df5ee 2023-04-27 03:55:30,946 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/ca60034b4805405c8eb5298207afdc92 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/ca60034b4805405c8eb5298207afdc92 2023-04-27 03:55:30,947 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/dfebd9c80ea44baab16e9ad9d53f5fab to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/dfebd9c80ea44baab16e9ad9d53f5fab 2023-04-27 03:55:30,949 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/8ba218714d5144f7be0831b85d3e2214, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/d59863b993da4474bfd9326b0c36bb51, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/c81b2403f04341f08ba1c494a8a5b5e2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/614c85b0efc04e189067c3a9bd4ba3e8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/5abe8125cada4ddb9a5959929aabc243, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/3219a1ca6b894c2cb3372d07413a01f9, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/f06f13eff0354fd4ae6d4b5d2fb4a4b9, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/a74fb38ab1e340f5a9ed80b2da779585, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/7504b76793e44df8af079aec034474b5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/cd2f6ce233e74c50804b2fd00e95cb75, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/9726a89c9b7b49bd9e016efaf6d27da5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/a802f0aea21b4c809edbc1442886670f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/ee74c2f3e74d4df5ac7d2d0ab888840c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/be45cda04b91443abe48dba1667a81bd, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/9b04c104c01144cca4cc77b931c5fa2c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/0a40116ccc1c4f13819167717e5ad5e6, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/9393dfd093c2403bb8551e16913defb4, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/5973df858e06420c842f3492dab2bcb9, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/7dfe0d7c9b7e4af68463de29784ef5fa, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/eee4ec52c36e42bbbfd777beb7db0c77, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/c8ba9c9a5cd14a0cb0b3f42dfa5b94b8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/ced9356217e54abe99f1e34c31cd9cf5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/7768ee29b5404c4ea71e0bfa2f6b6750] to archive 2023-04-27 03:55:30,950 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 03:55:30,951 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/8ba218714d5144f7be0831b85d3e2214 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/8ba218714d5144f7be0831b85d3e2214 2023-04-27 03:55:30,952 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/d59863b993da4474bfd9326b0c36bb51 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/d59863b993da4474bfd9326b0c36bb51 2023-04-27 03:55:30,954 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/c81b2403f04341f08ba1c494a8a5b5e2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/c81b2403f04341f08ba1c494a8a5b5e2 2023-04-27 03:55:30,954 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/614c85b0efc04e189067c3a9bd4ba3e8 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/614c85b0efc04e189067c3a9bd4ba3e8 2023-04-27 03:55:30,955 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/5abe8125cada4ddb9a5959929aabc243 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/5abe8125cada4ddb9a5959929aabc243 2023-04-27 03:55:30,956 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/3219a1ca6b894c2cb3372d07413a01f9 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/3219a1ca6b894c2cb3372d07413a01f9 2023-04-27 03:55:30,957 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/f06f13eff0354fd4ae6d4b5d2fb4a4b9 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/f06f13eff0354fd4ae6d4b5d2fb4a4b9 2023-04-27 03:55:30,957 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/a74fb38ab1e340f5a9ed80b2da779585 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/a74fb38ab1e340f5a9ed80b2da779585 2023-04-27 03:55:30,958 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/7504b76793e44df8af079aec034474b5 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/7504b76793e44df8af079aec034474b5 2023-04-27 03:55:30,959 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/cd2f6ce233e74c50804b2fd00e95cb75 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/cd2f6ce233e74c50804b2fd00e95cb75 2023-04-27 03:55:30,960 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/9726a89c9b7b49bd9e016efaf6d27da5 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/9726a89c9b7b49bd9e016efaf6d27da5 2023-04-27 03:55:30,960 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/a802f0aea21b4c809edbc1442886670f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/a802f0aea21b4c809edbc1442886670f 2023-04-27 03:55:30,961 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/ee74c2f3e74d4df5ac7d2d0ab888840c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/ee74c2f3e74d4df5ac7d2d0ab888840c 2023-04-27 03:55:30,962 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/be45cda04b91443abe48dba1667a81bd to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/be45cda04b91443abe48dba1667a81bd 2023-04-27 03:55:30,963 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/9b04c104c01144cca4cc77b931c5fa2c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/9b04c104c01144cca4cc77b931c5fa2c 2023-04-27 03:55:30,963 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/0a40116ccc1c4f13819167717e5ad5e6 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/0a40116ccc1c4f13819167717e5ad5e6 2023-04-27 03:55:30,964 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/9393dfd093c2403bb8551e16913defb4 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/9393dfd093c2403bb8551e16913defb4 2023-04-27 03:55:30,965 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/5973df858e06420c842f3492dab2bcb9 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/5973df858e06420c842f3492dab2bcb9 2023-04-27 03:55:30,966 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/7dfe0d7c9b7e4af68463de29784ef5fa to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/7dfe0d7c9b7e4af68463de29784ef5fa 2023-04-27 03:55:30,966 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/eee4ec52c36e42bbbfd777beb7db0c77 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/eee4ec52c36e42bbbfd777beb7db0c77 2023-04-27 03:55:30,967 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/c8ba9c9a5cd14a0cb0b3f42dfa5b94b8 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/c8ba9c9a5cd14a0cb0b3f42dfa5b94b8 2023-04-27 03:55:30,968 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/ced9356217e54abe99f1e34c31cd9cf5 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/ced9356217e54abe99f1e34c31cd9cf5 2023-04-27 03:55:30,969 DEBUG [StoreCloser-TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/7768ee29b5404c4ea71e0bfa2f6b6750 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/7768ee29b5404c4ea71e0bfa2f6b6750 2023-04-27 03:55:30,972 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/recovered.edits/489.seqid, newMaxSeqId=489, maxSeqId=4 2023-04-27 03:55:30,973 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053. 2023-04-27 03:55:30,974 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e0314b11a1068d19a1052690701d1053: 2023-04-27 03:55:30,975 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:30,975 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=e0314b11a1068d19a1052690701d1053, regionState=CLOSED 2023-04-27 03:55:30,975 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567730975"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682567730975"}]},"ts":"1682567730975"} 2023-04-27 03:55:30,977 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-27 03:55:30,977 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure e0314b11a1068d19a1052690701d1053, server=jenkins-hbase4.apache.org,40885,1682567618305 in 2.7930 sec 2023-04-27 03:55:30,978 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-27 03:55:30,978 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e0314b11a1068d19a1052690701d1053, UNASSIGN in 2.7970 sec 2023-04-27 03:55:30,979 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567730979"}]},"ts":"1682567730979"} 2023-04-27 03:55:30,980 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-27 03:55:30,981 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-27 03:55:30,982 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.8100 sec 2023-04-27 03:55:32,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-27 03:55:32,282 INFO [Listener at localhost/33933] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-27 03:55:32,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-27 03:55:32,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:55:32,285 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:55:32,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-27 03:55:32,285 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:55:32,290 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:32,291 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A, FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B, FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C, FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/recovered.edits] 2023-04-27 03:55:32,294 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/b7846695d2144130bcb67e23a0bdeb91 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/b7846695d2144130bcb67e23a0bdeb91 2023-04-27 03:55:32,294 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/f515f9916124477dad4a2bc5e08ece8d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/A/f515f9916124477dad4a2bc5e08ece8d 2023-04-27 03:55:32,297 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/1e12314615b44c6796f744f6947adc6e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/1e12314615b44c6796f744f6947adc6e 2023-04-27 03:55:32,298 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/efa76fc2ea4d42c487598921cb638181 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/B/efa76fc2ea4d42c487598921cb638181 2023-04-27 03:55:32,300 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/127d372d665e4ddf99bd7f349d8ae504 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/127d372d665e4ddf99bd7f349d8ae504 2023-04-27 03:55:32,301 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/1f4b55ca232848d6a1246450a32d03fc to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/1f4b55ca232848d6a1246450a32d03fc 2023-04-27 03:55:32,301 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/24b8557fada04823ab7df4c2179877d9 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/24b8557fada04823ab7df4c2179877d9 2023-04-27 03:55:32,302 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/459fa33d82e04fbfb37ac53ef30e501a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/459fa33d82e04fbfb37ac53ef30e501a 2023-04-27 03:55:32,303 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/5882d39adba74b019988488ba89e1ee9 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/5882d39adba74b019988488ba89e1ee9 2023-04-27 03:55:32,304 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/5e3d80b2f490430e977514e5aef153be to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/5e3d80b2f490430e977514e5aef153be 2023-04-27 03:55:32,305 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/d713bc0074234d65888e6432c3b7c9dd to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/C/d713bc0074234d65888e6432c3b7c9dd 2023-04-27 03:55:32,307 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/recovered.edits/489.seqid to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053/recovered.edits/489.seqid 2023-04-27 03:55:32,307 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/e0314b11a1068d19a1052690701d1053 2023-04-27 03:55:32,308 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-27 03:55:32,308 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-27 03:55:32,308 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-27 03:55:32,311 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304271ebcc9b70b51476387480c319c898fdc to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304271ebcc9b70b51476387480c319c898fdc 2023-04-27 03:55:32,312 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272251a8a1bc1a47f38d1c3313e9bb8b00 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272251a8a1bc1a47f38d1c3313e9bb8b00 2023-04-27 03:55:32,313 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272cda16b072bd4e6bb056f2b93de7f792 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272cda16b072bd4e6bb056f2b93de7f792 2023-04-27 03:55:32,314 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272d1e1a62adb14a85b5242e349ac8dba9 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272d1e1a62adb14a85b5242e349ac8dba9 2023-04-27 03:55:32,315 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272eb01b5441ea4d958feb151449272fef to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272eb01b5441ea4d958feb151449272fef 2023-04-27 03:55:32,316 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427356830f9ba1449378c7fd90c43401aec to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427356830f9ba1449378c7fd90c43401aec 2023-04-27 03:55:32,317 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304274743b2c3c8284211839dd2b5a6867809 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304274743b2c3c8284211839dd2b5a6867809 2023-04-27 03:55:32,318 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304274bc82964c1594c56837a6ca0ec73cf72 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304274bc82964c1594c56837a6ca0ec73cf72 2023-04-27 03:55:32,319 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304274d0c380bc8344d3f8e6d8eaf9afe0dad to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304274d0c380bc8344d3f8e6d8eaf9afe0dad 2023-04-27 03:55:32,320 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427536adcfa85db4b489c3a90042b4c6543 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427536adcfa85db4b489c3a90042b4c6543 2023-04-27 03:55:32,321 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304275db9e5676d0045fa98fa73862b3bedb9 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304275db9e5676d0045fa98fa73862b3bedb9 2023-04-27 03:55:32,322 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304275fc765eab8274000b8275a36be559aa6 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304275fc765eab8274000b8275a36be559aa6 2023-04-27 03:55:32,323 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427805ff193ca52451ab188635e2ba5ebee to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427805ff193ca52451ab188635e2ba5ebee 2023-04-27 03:55:32,325 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304278d45a4563de149e79208b9cfa4de6aaf to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304278d45a4563de149e79208b9cfa4de6aaf 2023-04-27 03:55:32,326 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427a0b70bb953d7443eacd72dea645403e4 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427a0b70bb953d7443eacd72dea645403e4 2023-04-27 03:55:32,328 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427a45a7f4ad0b1466089a3508930741523 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427a45a7f4ad0b1466089a3508930741523 2023-04-27 03:55:32,329 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427a65bc443f06347e8848679e04416766a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427a65bc443f06347e8848679e04416766a 2023-04-27 03:55:32,330 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427abb555027a4b450da8906155d94f2dfb to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427abb555027a4b450da8906155d94f2dfb 2023-04-27 03:55:32,331 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427aee275071b504f6c90ea08103eea0009 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427aee275071b504f6c90ea08103eea0009 2023-04-27 03:55:32,332 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427b86597710b9a4d8e8ac13647bee6a5f7 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427b86597710b9a4d8e8ac13647bee6a5f7 2023-04-27 03:55:32,333 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427eb6fffdf6caa4a929fe28da11b276001 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427eb6fffdf6caa4a929fe28da11b276001 2023-04-27 03:55:32,335 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427f26fee1ef3084545bc32162c2524e728 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427f26fee1ef3084545bc32162c2524e728 2023-04-27 03:55:32,335 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427f65e6e43f54a48809580d8c03e70809f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427f65e6e43f54a48809580d8c03e70809f 2023-04-27 03:55:32,337 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427f8055d443bf5469097ea7239dbeff605 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427f8055d443bf5469097ea7239dbeff605 2023-04-27 03:55:32,337 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-27 03:55:32,339 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:55:32,341 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-27 03:55:32,342 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-27 03:55:32,343 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:55:32,343 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-27 03:55:32,343 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682567732343"}]},"ts":"9223372036854775807"} 2023-04-27 03:55:32,345 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-27 03:55:32,345 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => e0314b11a1068d19a1052690701d1053, NAME => 'TestAcidGuarantees,,1682567702096.e0314b11a1068d19a1052690701d1053.', STARTKEY => '', ENDKEY => ''}] 2023-04-27 03:55:32,345 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-27 03:55:32,345 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682567732345"}]},"ts":"9223372036854775807"} 2023-04-27 03:55:32,346 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-27 03:55:32,348 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:55:32,349 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 65 msec 2023-04-27 03:55:32,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-27 03:55:32,386 INFO [Listener at localhost/33933] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-27 03:55:32,398 INFO [Listener at localhost/33933] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=224 (was 230), OpenFileDescriptor=531 (was 555), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=306 (was 346), ProcessCount=170 (was 172), AvailableMemoryMB=11876 (was 11955) 2023-04-27 03:55:32,408 INFO [Listener at localhost/33933] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=224, OpenFileDescriptor=531, MaxFileDescriptor=60000, SystemLoadAverage=306, ProcessCount=170, AvailableMemoryMB=11875 2023-04-27 03:55:32,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-27 03:55:32,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-27 03:55:32,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-27 03:55:32,412 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-27 03:55:32,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-27 03:55:32,413 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-27 03:55:32,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-27 03:55:32,414 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:32,414 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4 empty. 2023-04-27 03:55:32,415 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:32,415 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-27 03:55:32,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-27 03:55:32,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-27 03:55:32,821 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-27 03:55:32,823 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 98cbc3f1a78abce9e99d0349272cd6a4, NAME => 'TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp 2023-04-27 03:55:33,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-27 03:55:33,228 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 03:55:33,229 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 98cbc3f1a78abce9e99d0349272cd6a4, disabling compactions & flushes 2023-04-27 03:55:33,229 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:33,229 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:33,229 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. after waiting 0 ms 2023-04-27 03:55:33,229 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:33,229 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:33,229 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:33,231 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-27 03:55:33,231 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567733231"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682567733231"}]},"ts":"1682567733231"} 2023-04-27 03:55:33,232 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-27 03:55:33,233 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-27 03:55:33,233 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567733233"}]},"ts":"1682567733233"} 2023-04-27 03:55:33,234 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-27 03:55:33,240 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=98cbc3f1a78abce9e99d0349272cd6a4, ASSIGN}] 2023-04-27 03:55:33,241 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=98cbc3f1a78abce9e99d0349272cd6a4, ASSIGN 2023-04-27 03:55:33,242 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=98cbc3f1a78abce9e99d0349272cd6a4, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40885,1682567618305; forceNewPlan=false, retain=false 2023-04-27 03:55:33,393 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=98cbc3f1a78abce9e99d0349272cd6a4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:33,393 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567733393"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682567733393"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682567733393"}]},"ts":"1682567733393"} 2023-04-27 03:55:33,394 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305}] 2023-04-27 03:55:33,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-27 03:55:33,548 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:33,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 98cbc3f1a78abce9e99d0349272cd6a4, NAME => 'TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.', STARTKEY => '', ENDKEY => ''} 2023-04-27 03:55:33,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:33,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 03:55:33,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:33,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:33,550 INFO [StoreOpener-98cbc3f1a78abce9e99d0349272cd6a4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:33,551 DEBUG [StoreOpener-98cbc3f1a78abce9e99d0349272cd6a4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A 2023-04-27 03:55:33,551 DEBUG [StoreOpener-98cbc3f1a78abce9e99d0349272cd6a4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A 2023-04-27 03:55:33,552 INFO [StoreOpener-98cbc3f1a78abce9e99d0349272cd6a4-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:55:33,552 INFO [StoreOpener-98cbc3f1a78abce9e99d0349272cd6a4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 98cbc3f1a78abce9e99d0349272cd6a4 columnFamilyName A 2023-04-27 03:55:33,553 INFO [StoreOpener-98cbc3f1a78abce9e99d0349272cd6a4-1] regionserver.HStore(310): Store=98cbc3f1a78abce9e99d0349272cd6a4/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:55:33,553 INFO [StoreOpener-98cbc3f1a78abce9e99d0349272cd6a4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:33,554 DEBUG [StoreOpener-98cbc3f1a78abce9e99d0349272cd6a4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B 2023-04-27 03:55:33,554 DEBUG [StoreOpener-98cbc3f1a78abce9e99d0349272cd6a4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B 2023-04-27 03:55:33,554 INFO [StoreOpener-98cbc3f1a78abce9e99d0349272cd6a4-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:55:33,555 INFO [StoreOpener-98cbc3f1a78abce9e99d0349272cd6a4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 98cbc3f1a78abce9e99d0349272cd6a4 columnFamilyName B 2023-04-27 03:55:33,555 INFO [StoreOpener-98cbc3f1a78abce9e99d0349272cd6a4-1] regionserver.HStore(310): Store=98cbc3f1a78abce9e99d0349272cd6a4/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:55:33,555 INFO [StoreOpener-98cbc3f1a78abce9e99d0349272cd6a4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:33,556 DEBUG [StoreOpener-98cbc3f1a78abce9e99d0349272cd6a4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C 2023-04-27 03:55:33,556 DEBUG [StoreOpener-98cbc3f1a78abce9e99d0349272cd6a4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C 2023-04-27 03:55:33,556 INFO [StoreOpener-98cbc3f1a78abce9e99d0349272cd6a4-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:55:33,556 INFO [StoreOpener-98cbc3f1a78abce9e99d0349272cd6a4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 98cbc3f1a78abce9e99d0349272cd6a4 columnFamilyName C 2023-04-27 03:55:33,557 INFO [StoreOpener-98cbc3f1a78abce9e99d0349272cd6a4-1] regionserver.HStore(310): Store=98cbc3f1a78abce9e99d0349272cd6a4/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:55:33,557 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:33,558 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:33,558 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:33,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-27 03:55:33,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:33,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-27 03:55:33,563 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 98cbc3f1a78abce9e99d0349272cd6a4; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60226697, jitterRate=-0.10255227982997894}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-27 03:55:33,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:33,563 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., pid=49, masterSystemTime=1682567733545 2023-04-27 03:55:33,564 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:33,564 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:33,565 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=98cbc3f1a78abce9e99d0349272cd6a4, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:33,565 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567733565"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682567733565"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682567733565"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682567733565"}]},"ts":"1682567733565"} 2023-04-27 03:55:33,566 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-27 03:55:33,566 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 in 171 msec 2023-04-27 03:55:33,567 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-27 03:55:33,568 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=98cbc3f1a78abce9e99d0349272cd6a4, ASSIGN in 326 msec 2023-04-27 03:55:33,568 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-27 03:55:33,568 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567733568"}]},"ts":"1682567733568"} 2023-04-27 03:55:33,569 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-27 03:55:33,571 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-27 03:55:33,571 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1610 sec 2023-04-27 03:55:34,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-27 03:55:34,517 INFO [Listener at localhost/33933] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-27 03:55:34,518 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x37705401 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:55:34,523 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ec943a9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:55:34,524 DEBUG [Listener at localhost/33933] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:55:34,526 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45910, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:55:34,527 DEBUG [Listener at localhost/33933] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-27 03:55:34,529 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35944, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-27 03:55:34,530 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x15880ffb to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:55:34,534 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20ed9117, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:55:34,534 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x052203cb to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:55:34,539 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50002343, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:55:34,539 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x420f945a to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:55:34,543 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5726fb8b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:55:34,543 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x51b8df2a to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:55:34,548 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f2b3c03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:55:34,548 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x67346c47 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:55:34,551 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70e2be04, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:55:34,552 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x067ea274 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:55:34,558 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63093e21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:55:34,558 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x78592df0 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:55:34,563 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b475d8d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:55:34,564 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x659c68ab to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:55:34,580 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@549bef45, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:55:34,580 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x680e773e to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:55:34,588 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ed183a4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:55:34,588 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x0b8f652e to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:55:34,600 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10a344e6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:55:34,608 DEBUG [hconnection-0x23beb04c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:55:34,608 DEBUG [hconnection-0x175f9c5c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:55:34,609 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45930, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:55:34,616 DEBUG [hconnection-0x695e6653-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:55:34,621 DEBUG [hconnection-0x1acee742-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:55:34,621 DEBUG [hconnection-0x69263146-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:55:34,622 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45936, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:55:34,625 DEBUG [hconnection-0x3130137e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:55:34,625 DEBUG [hconnection-0x75a43ecd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:55:34,625 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45932, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:55:34,625 DEBUG [hconnection-0x1be6a9a0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:55:34,625 DEBUG [hconnection-0x4e9ba2e9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:55:34,629 DEBUG [hconnection-0x451cefd5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:55:34,629 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45964, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:55:34,631 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45948, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:55:34,631 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45914, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:55:34,631 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45978, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:55:34,631 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45934, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:55:34,632 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45952, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:55:34,633 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45982, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:55:34,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:34,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:34,653 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-27 03:55:34,653 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:34,653 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:34,653 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:34,653 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:34,653 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:34,653 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:34,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:34,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:34,656 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:34,656 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:34,656 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:34,656 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:34,656 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,657 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:34,658 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:34,658 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:34,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:34,658 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,658 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:34,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:34,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:34,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:34,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:34,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:34,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:34,663 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:34,663 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:34,663 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:34,663 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:34,663 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:34,664 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:34,664 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:34,664 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:34,664 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:34,664 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:34,664 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:34,664 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,664 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:34,664 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:34,671 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:34,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:34,671 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:34,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:34,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:34,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:34,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:34,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:34,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42d8e17f[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:34,674 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:34,674 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:34,675 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:34,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:34,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:34,676 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,676 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:34,676 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:34,676 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:34,676 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:34,676 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:34,676 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:34,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:34,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567794677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:34,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567794678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:34,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1682567734672, max timestamp=1682567734678Num uniques 859; 2023-04-27 03:55:34,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1682567734672, max timestamp=1682567734678Num uniques 859; 2023-04-27 03:55:34,680 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,680 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:34,679 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:34,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1682567734672, max timestamp=1682567734678Num uniques 859; 2023-04-27 03:55:34,681 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:34,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-27 03:55:34,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-27 03:55:34,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567794679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,681 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:34,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-27 03:55:34,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567794678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:34,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:34,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:34,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:34,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:34,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:34,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:34,685 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:34,686 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:34,686 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:34,686 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:34,686 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:34,688 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:34,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:34,688 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,688 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:34,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:34,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:34,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:34,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,689 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:34,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:34,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:34,691 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:34,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:34,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,694 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:34,694 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:34,694 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:34,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:34,694 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:34,694 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:34,694 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,694 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:34,694 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:34,694 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:34,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:34,694 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:34,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:34,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:34,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:34,695 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:34,695 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:34,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:34,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567794703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:34,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567794781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:34,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:34,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567794782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567794783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:34,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567794786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,795 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:34,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:34,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:34,806 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:34,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567794804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:34,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:34,812 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:34,812 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:34,812 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:34,812 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:34,813 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,813 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:34,814 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,814 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:34,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:34,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:34,815 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,815 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:34,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:34,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:34,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:34,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:34,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:34,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:34,820 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:34,820 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:34,820 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:34,827 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:34,827 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:34,827 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:34,828 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:34,828 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:34,828 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:34,828 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:34,828 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:34,828 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,832 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:34,832 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:34,834 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,834 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:34,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:34,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:34,835 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:34,835 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:34,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:34,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:34,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:34,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:34,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:34,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2717c53[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:34,836 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:34,836 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:34,838 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:34,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:34,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:34,838 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:34,838 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:34,838 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:34,839 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,839 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:34,839 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:34,840 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:34,840 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:34,840 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:34,840 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:34,840 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:34,840 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:34,840 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:34,840 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:34,843 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,843 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:34,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,843 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:34,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,844 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:34,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:34,845 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:34,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:34,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,848 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:34,848 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:34,848 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,849 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:34,849 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:34,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:34,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:34,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:34,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:34,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:34,849 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:34,849 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,849 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:34,849 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:34,849 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,849 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:34,849 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:34,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:34,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:34,949 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:34,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:34,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:34,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:34,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:34,960 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:34,960 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:34,960 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:34,960 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:34,961 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,961 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:34,962 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:34,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:34,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:34,962 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:34,963 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,963 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:34,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:34,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:34,964 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:34,965 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:34,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:34,965 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:34,965 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:34,965 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:34,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:34,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:34,968 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:34,968 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:34,968 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:34,968 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:34,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:34,969 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:34,969 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,969 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:34,969 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:34,970 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,970 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:34,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:34,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:34,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:34,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:34,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:34,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35f81377[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:34,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:34,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:34,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:34,984 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:34,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567794984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567794984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,985 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:34,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:34,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:34,986 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,986 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:34,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567794985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,986 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:34,986 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:34,986 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:34,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:34,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567794988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,992 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:34,992 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:34,992 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,992 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:34,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:34,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:34,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:34,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:34,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:34,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:34,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:34,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:34,995 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:34,995 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:34,995 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:34,995 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:34,995 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:34,997 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,997 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:34,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,997 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:34,997 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:34,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:34,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:34,999 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:34,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:35,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,002 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:35,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:35,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:35,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:35,002 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:35,002 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:35,003 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,003 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,003 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,003 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,003 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,003 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,003 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,003 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:35,003 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:35,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:35,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:35,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:35,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:35,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:35,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567795008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,102 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:35,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:35,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:35,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:35,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:35,124 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/340996cbeed044d7967aafbf4bd8b0b1 2023-04-27 03:55:35,124 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:35,124 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:35,124 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:35,124 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:35,125 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,125 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,127 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:35,127 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:35,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:35,127 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,127 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:35,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:35,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:35,131 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:35,131 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:35,131 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:35,131 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:35,131 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:35,135 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:35,135 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:35,136 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:35,136 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:35,136 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:35,136 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:35,142 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,142 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:35,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:35,142 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,142 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:35,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:35,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:35,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cb55018[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:35,147 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:35,147 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,148 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,148 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:35,149 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,149 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:35,149 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:35,149 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:35,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:35,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:35,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:35,151 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:35,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:35,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:35,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:35,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:35,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:35,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:35,156 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:35,156 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:35,156 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:35,156 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:35,156 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:35,159 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,159 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:35,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,159 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:35,160 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:35,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:35,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:35,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:35,162 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:35,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,166 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,166 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,166 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:35,166 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:35,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:35,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:35,166 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:35,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:35,166 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,166 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:35,166 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:35,166 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,166 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:35,167 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:35,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:35,167 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:35,267 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:35,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:35,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:35,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:35,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:35,272 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:35,272 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:35,272 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:35,272 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:35,272 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,272 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,274 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,274 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:35,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:35,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:35,274 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,274 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:35,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:35,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:35,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:35,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:35,275 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:35,275 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:35,275 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:35,275 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:35,275 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:35,276 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:35,276 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:35,276 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:35,276 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:35,276 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:35,276 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:35,276 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,276 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:35,276 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:35,279 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,279 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,279 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,279 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:35,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:35,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:35,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:35,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:35,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:35,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:35,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59e391e8[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:35,287 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:35,287 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,289 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,289 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:35,289 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:35,289 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:35,289 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,289 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:35,290 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:35,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567795286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,292 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:35,292 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:35,292 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,292 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:35,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:35,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:35,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:35,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:35,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:35,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:35,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:35,295 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:35,295 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:35,295 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:35,295 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:35,295 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:35,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:35,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567795288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:35,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567795289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:35,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567795291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,298 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,298 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:35,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,298 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:35,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:35,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:35,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:35,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:35,300 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:35,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,301 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:35,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,303 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,303 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:35,303 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:35,303 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,303 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,303 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:35,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:35,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:35,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:35,305 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,305 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,305 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,305 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:35,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:35,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:35,305 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:35,305 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:35,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:35,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567795311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,405 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:35,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:35,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:35,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:35,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:35,409 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:35,409 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:35,410 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:35,410 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:35,410 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,410 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,411 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,411 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,412 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:35,412 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:35,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:35,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:35,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:35,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:35,413 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:35,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:35,413 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:35,413 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:35,413 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:35,414 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:35,414 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:35,414 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:35,414 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:35,414 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:35,414 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:35,414 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:35,414 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:35,414 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,414 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:35,414 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:35,417 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,417 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:35,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:35,417 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,417 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:35,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:35,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:35,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:35,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:35,419 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:35,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5079fd63[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:35,419 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,420 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,420 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:35,420 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:35,420 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:35,420 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,420 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:35,422 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,422 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:35,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:35,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:35,422 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:35,422 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:35,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:35,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:35,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:35,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:35,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:35,426 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:35,426 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:35,426 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:35,426 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:35,426 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:35,427 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,427 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:35,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:35,428 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:35,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:35,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:35,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:35,428 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:35,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,429 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:35,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,435 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,435 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:35,435 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:35,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:35,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:35,435 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:35,435 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:35,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:35,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:35,436 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,436 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:35,436 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:35,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:35,436 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,436 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:35,436 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,436 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,536 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:35,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:35,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:35,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:35,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:35,540 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:35,540 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:35,540 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:35,540 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:35,540 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,540 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,542 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,542 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:35,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:35,542 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,542 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:35,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:35,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:35,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:35,543 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:35,543 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:35,543 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:35,543 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:35,543 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:35,544 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:35,544 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:35,544 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:35,544 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:35,544 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:35,544 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:35,544 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,544 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:35,544 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:35,546 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:35,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:35,547 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:35,547 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,547 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:35,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:35,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:35,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:35,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25ef2fb0[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:35,548 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:35,548 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,549 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,550 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:35,550 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,550 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:35,550 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:35,550 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:35,552 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:35,552 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:35,552 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:35,552 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:35,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:35,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:35,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:35,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:35,554 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/5f717233d77c49cd908915e38a25776d 2023-04-27 03:55:35,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:35,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:35,556 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:35,556 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:35,556 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:35,556 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:35,556 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:35,559 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,559 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,560 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:35,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:35,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:35,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:35,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:35,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:35,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,561 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:35,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,563 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:35,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,566 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,567 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,567 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:35,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:35,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:35,567 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:35,567 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:35,567 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:35,567 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:35,567 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,567 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,567 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,567 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:35,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:35,567 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:35,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:35,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:35,667 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:35,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:35,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:35,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:35,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:35,670 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:35,670 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:35,671 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:35,671 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:35,671 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,671 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,672 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:35,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:35,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:35,672 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,673 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,673 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:35,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:35,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:35,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:35,674 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:35,674 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:35,674 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:35,674 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:35,674 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:35,674 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:35,674 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:35,674 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:35,674 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:35,675 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:35,675 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:35,675 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,675 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:35,675 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:35,677 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,677 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:35,678 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,678 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:35,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:35,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:35,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:35,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:35,679 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:35,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@532c8e85[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:35,679 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,680 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,681 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:35,681 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:35,681 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:35,681 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,681 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:35,682 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,682 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:35,682 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:35,682 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:35,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:35,686 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:35,687 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:35,687 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:35,687 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:35,687 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:35,688 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,688 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:35,688 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:35,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:35,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:35,689 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:35,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,692 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:35,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,703 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:35,703 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:35,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:35,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:35,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:35,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:35,703 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,703 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,703 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,703 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:35,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:35,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:35,703 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,704 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,704 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,704 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,704 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:35,704 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:35,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:35,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567795792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:35,804 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:35,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567795798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:35,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567795798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:35,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567795799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:35,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:35,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:35,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:35,808 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:35,808 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:35,808 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:35,808 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:35,810 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,810 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,812 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,812 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:35,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:35,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:35,812 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,812 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:35,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:35,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:35,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:35,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:35,814 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:35,814 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:35,814 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:35,814 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:35,814 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:35,814 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:35,814 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:35,815 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:35,815 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:35,815 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:35,815 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:35,815 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,815 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:35,815 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:35,817 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,817 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:35,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:35,818 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,818 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:35,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:35,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@168529d9[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:35,821 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:35,821 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,823 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,823 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:35,823 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:35,823 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:35,823 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,823 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:35,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:35,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567795820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,826 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,826 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:35,826 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:35,826 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:35,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:35,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:35,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:35,832 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:35,833 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:35,833 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:35,833 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:35,833 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:35,834 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,834 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:35,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:35,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,834 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:35,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:35,835 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,836 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:35,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,847 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,848 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,848 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:35,848 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,848 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,848 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,848 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,848 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:35,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:35,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:35,848 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:35,848 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:35,849 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:35,849 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:35,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:35,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:35,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:35,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:35,948 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:35,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:35,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:35,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:35,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:35,953 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:35,953 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:35,953 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:35,953 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:35,953 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,953 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,955 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:35,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:35,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:35,955 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,956 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,956 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:35,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:35,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,957 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:35,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:35,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:35,958 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:35,958 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:35,958 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:35,958 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:35,958 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:35,958 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:35,958 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:35,958 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:35,958 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:35,958 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:35,958 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:35,958 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,958 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:35,958 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:35,960 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,960 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:35,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:35,960 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,960 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:35,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:35,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:35,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:35,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:35,962 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:35,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9085c8f[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:35,962 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,964 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,964 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:35,964 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:35,964 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:35,964 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,964 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:35,966 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:35,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:35,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:35,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:35,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,967 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:35,967 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:35,967 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:35,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:35,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:35,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:35,968 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:35,968 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:35,968 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:35,968 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:35,968 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:35,969 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/402f32b9f7cd4fcb970cf24d47e9cf1a 2023-04-27 03:55:35,970 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:35,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:35,970 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,970 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:35,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:35,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:35,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:35,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:35,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:35,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:35,973 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,976 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/340996cbeed044d7967aafbf4bd8b0b1 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/340996cbeed044d7967aafbf4bd8b0b1 2023-04-27 03:55:35,977 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,978 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,978 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:35,978 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:35,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:35,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:35,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:35,978 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:35,978 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:35,978 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,978 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:35,978 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:35,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:35,978 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:35,978 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:35,978 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:35,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:35,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:35,980 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/340996cbeed044d7967aafbf4bd8b0b1, entries=150, sequenceid=13, filesize=11.6 K 2023-04-27 03:55:35,981 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/5f717233d77c49cd908915e38a25776d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/5f717233d77c49cd908915e38a25776d 2023-04-27 03:55:35,984 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/5f717233d77c49cd908915e38a25776d, entries=150, sequenceid=13, filesize=11.6 K 2023-04-27 03:55:35,985 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/402f32b9f7cd4fcb970cf24d47e9cf1a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/402f32b9f7cd4fcb970cf24d47e9cf1a 2023-04-27 03:55:35,989 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/402f32b9f7cd4fcb970cf24d47e9cf1a, entries=150, sequenceid=13, filesize=11.6 K 2023-04-27 03:55:35,990 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for 98cbc3f1a78abce9e99d0349272cd6a4 in 1337ms, sequenceid=13, compaction requested=false 2023-04-27 03:55:35,990 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:36,078 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:36,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:36,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:36,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:36,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:36,081 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:36,081 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:36,081 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:36,081 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:36,082 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,082 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:36,084 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:36,084 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:36,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:36,084 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,084 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:36,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:36,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:36,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:36,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:36,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:36,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:36,085 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:36,085 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:36,085 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:36,085 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:36,085 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:36,085 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:36,085 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:36,085 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=127.47 KB heapSize=341.63 KB 2023-04-27 03:55:36,086 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:36,086 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:36,086 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:36,086 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:36,086 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:36,086 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:36,099 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=44 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/e9a8dc6e26f646e788f6576b56d628ea 2023-04-27 03:55:36,108 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=44 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/0acae6104d8645cbb87c450b5608dc1f 2023-04-27 03:55:36,504 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-27 03:55:36,517 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=44 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/c386f0ca13eb4b669ad7738398aa9922 2023-04-27 03:55:36,521 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/e9a8dc6e26f646e788f6576b56d628ea as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/e9a8dc6e26f646e788f6576b56d628ea 2023-04-27 03:55:36,525 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/e9a8dc6e26f646e788f6576b56d628ea, entries=150, sequenceid=44, filesize=11.6 K 2023-04-27 03:55:36,527 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/0acae6104d8645cbb87c450b5608dc1f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/0acae6104d8645cbb87c450b5608dc1f 2023-04-27 03:55:36,530 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/0acae6104d8645cbb87c450b5608dc1f, entries=150, sequenceid=44, filesize=11.6 K 2023-04-27 03:55:36,531 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/c386f0ca13eb4b669ad7738398aa9922 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/c386f0ca13eb4b669ad7738398aa9922 2023-04-27 03:55:36,546 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/c386f0ca13eb4b669ad7738398aa9922, entries=150, sequenceid=44, filesize=11.6 K 2023-04-27 03:55:36,546 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~341.58 KB/349776, currentSize=0 B/0 for 98cbc3f1a78abce9e99d0349272cd6a4 in 461ms, sequenceid=44, compaction requested=false 2023-04-27 03:55:36,547 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:36,547 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:36,547 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:36,547 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:36,547 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,551 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:36,551 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:36,553 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,553 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:36,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:36,553 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,553 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:36,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:36,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:36,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:36,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:36,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:36,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55c660b1[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:36,555 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:36,555 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,556 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,556 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:36,556 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:36,556 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,556 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:36,556 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:36,558 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:36,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:36,558 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:36,558 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:36,558 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:36,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:36,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:36,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:36,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:36,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:36,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:36,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:36,563 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:36,563 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:36,563 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:36,563 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:36,563 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:36,568 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,568 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:36,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:36,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,568 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:36,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:36,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:36,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:36,569 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:36,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,570 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:36,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,574 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,574 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:36,574 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:36,574 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,574 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,574 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,574 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,574 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:36,574 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:36,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:36,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:36,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:36,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:36,574 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:36,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:36,575 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:36,575 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:36,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:36,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:36,675 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:36,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:36,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:36,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:36,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:36,692 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:36,692 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:36,692 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:36,692 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:36,693 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,693 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:36,694 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:36,694 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:36,695 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,695 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:36,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:36,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:36,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:36,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:36,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:36,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:36,710 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:36,710 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:36,710 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:36,714 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:36,715 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:36,715 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:36,715 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:36,715 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:36,715 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:36,715 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:36,715 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:36,715 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,715 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:36,716 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:36,717 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,717 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:36,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:36,717 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,717 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:36,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:36,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:36,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:36,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:36,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:36,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6229942a[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:36,719 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:36,719 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,721 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:36,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:36,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:36,722 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,722 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:36,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:36,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:36,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:36,723 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:36,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:36,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:36,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:36,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:36,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:36,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:36,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:36,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:36,725 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:36,725 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:36,725 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:36,725 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:36,725 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:36,727 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,727 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,727 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:36,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:36,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:36,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:36,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:36,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:36,728 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:36,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:36,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:36,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:36,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:36,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,731 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,731 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:36,731 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:36,731 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:36,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:36,732 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:36,732 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:36,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:36,731 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,732 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,732 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:36,732 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:36,732 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:36,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:36,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:36,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:36,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:36,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:36,810 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:55:36,810 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:36,810 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:36,810 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:36,810 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:36,810 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:36,810 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:36,832 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:36,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:36,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:36,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:36,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:36,835 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:36,835 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:36,835 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:36,835 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:36,835 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,835 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:36,840 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:36,840 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:36,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:36,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:36,840 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,841 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:36,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:36,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:36,841 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:36,841 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:36,841 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:36,841 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:36,841 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:36,841 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:36,841 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:36,841 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:36,841 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:36,842 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:36,842 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:36,842 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:36,842 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:36,842 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:36,842 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,842 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:36,842 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:36,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567736835, max timestamp=1682567736842Num uniques 859; 2023-04-27 03:55:36,843 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:36,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:36,844 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567736835, max timestamp=1682567736842Num uniques 859; 2023-04-27 03:55:36,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:55:36,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:55:36,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567736835, max timestamp=1682567736842Num uniques 859; 2023-04-27 03:55:36,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:55:36,845 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,845 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:36,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:36,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:36,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:36,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:36,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:36,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ddda257[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:36,847 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:36,847 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,848 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:36,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:36,848 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:36,848 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,848 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:36,851 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,851 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:36,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:36,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:36,851 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:36,851 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:36,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:36,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:36,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:36,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:36,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:36,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:36,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:36,853 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:36,853 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:36,853 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:36,853 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:36,853 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:36,855 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,855 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:36,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:36,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,855 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:36,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:36,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:36,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:36,855 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:36,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:36,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:36,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,856 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:36,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:36,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,858 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,858 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:36,858 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:36,858 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:36,859 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:36,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:36,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:36,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:36,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:36,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:36,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,859 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:36,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,859 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:36,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:36,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:36,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:36,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567796862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:36,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567796863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:36,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567796864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:36,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567796869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:36,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567796870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,959 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:36,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:36,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:36,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:36,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:36,962 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:36,962 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:36,962 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:36,962 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:36,962 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,962 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:36,964 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,964 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:36,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:36,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:36,964 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,964 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:36,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:36,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:36,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:36,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:36,965 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:36,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:36,965 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:36,965 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:36,965 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:36,965 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:36,965 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:36,965 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:36,965 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:36,965 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:36,965 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:36,965 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:36,965 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:36,966 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,966 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:36,966 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:36,968 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:36,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:36,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,969 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:36,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:36,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:36,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:36,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:36,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:36,970 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:36,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@167a7147[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:36,970 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,971 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:36,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:36,971 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,971 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:36,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:36,972 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,973 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:36,973 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:36,973 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:36,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:36,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567796971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:36,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567796971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:36,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567796971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:36,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567796971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:36,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:36,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:36,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:36,976 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:36,976 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:36,976 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:36,976 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:36,976 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:36,977 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,977 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:36,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:36,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,977 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:36,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:36,978 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:36,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,979 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:36,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:36,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,982 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:36,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567796977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,982 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:36,982 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,982 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:36,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:36,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:36,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:36,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:36,982 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:36,982 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:36,982 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:36,982 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:36,983 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:36,983 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:36,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:36,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:36,983 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,983 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:36,983 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,083 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:37,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:37,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:37,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:37,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:37,086 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:37,086 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:37,086 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:37,086 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:37,086 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,086 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,087 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,087 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:37,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:37,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:37,088 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,088 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:37,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:37,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,089 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:37,089 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:37,089 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:37,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:37,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:37,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:37,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:37,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:37,089 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:37,089 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:37,089 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:37,089 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:37,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:37,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:37,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:37,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:37,091 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,091 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:37,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:37,091 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,091 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:37,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:37,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:37,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:37,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:37,092 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:37,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70df48d1[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:37,092 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,093 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,094 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:37,094 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:37,094 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,094 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:37,094 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:37,096 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,096 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:37,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:37,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:37,096 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:37,096 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:37,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:37,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:37,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:37,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:37,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:37,099 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:37,099 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:37,099 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:37,099 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:37,099 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:37,100 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,100 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:37,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:37,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,100 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:37,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:37,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:37,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:37,101 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:37,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,102 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:37,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,104 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:37,104 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:37,104 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,104 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:37,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:37,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:37,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:37,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:37,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:37,104 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,104 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,104 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,104 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:37,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:37,104 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,105 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,105 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:37,105 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:37,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:37,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567797174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:37,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567797175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:37,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567797175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:37,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567797176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:37,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567797182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,205 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:37,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:37,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:37,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:37,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:37,208 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:37,208 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:37,208 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:37,209 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:37,209 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,209 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,210 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:37,210 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:37,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:37,210 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,211 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:37,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:37,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:37,212 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:37,212 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:37,212 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:37,212 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:37,212 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:37,212 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:37,212 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:37,212 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:37,212 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:37,212 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:37,212 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:37,212 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:37,212 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:37,213 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,213 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:37,213 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:37,215 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,215 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:37,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:37,216 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,216 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:37,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:37,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:37,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:37,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:37,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ef91a24[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:37,217 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:37,217 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,218 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=55 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/8fb17fae54ab4d22892cf2642bba8a19 2023-04-27 03:55:37,221 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,221 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:37,221 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:37,221 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,221 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:37,221 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:37,223 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,223 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:37,223 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:37,223 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:37,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:37,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:37,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:37,225 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:37,225 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:37,225 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:37,225 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:37,225 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:37,227 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,227 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:37,227 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:37,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:37,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:37,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:37,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:37,229 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:37,233 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,237 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,237 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,237 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,237 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,237 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,237 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:37,237 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:37,237 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,237 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:37,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:37,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:37,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:37,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:37,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:37,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:37,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:37,240 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:37,240 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:37,340 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:37,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:37,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:37,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:37,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:37,343 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:37,343 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:37,343 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:37,343 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:37,344 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,344 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,345 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:37,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:37,345 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:37,346 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,346 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:37,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:37,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:37,347 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:37,347 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:37,347 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:37,347 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:37,347 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:37,347 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:37,347 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:37,347 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:37,347 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:37,347 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:37,347 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:37,347 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:37,347 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:37,347 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,347 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:37,347 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:37,349 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,349 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:37,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:37,349 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,349 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:37,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:37,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:37,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:37,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:37,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e06c4b6[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:37,350 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:37,350 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,351 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,352 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:37,352 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:37,352 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:37,352 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,352 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:37,354 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:37,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:37,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:37,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:37,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:37,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:37,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:37,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:37,360 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:37,360 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:37,360 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:37,360 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:37,360 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:37,363 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,363 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:37,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:37,363 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:37,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:37,364 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,365 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,367 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:37,367 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:37,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:37,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:37,367 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,367 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,367 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,367 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,367 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:37,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:37,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:37,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:37,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:37,367 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,368 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:37,368 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:37,368 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,368 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:37,468 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:37,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:37,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:37,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:37,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:37,471 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:37,471 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:37,471 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:37,471 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:37,472 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,472 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,473 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:37,473 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:37,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:37,473 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,473 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:37,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:37,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:37,474 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:37,474 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:37,474 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:37,474 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:37,474 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:37,474 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:37,474 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:37,474 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:37,474 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:37,474 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:37,474 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:37,474 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:37,475 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,475 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:37,475 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:37,477 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,477 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:37,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:37,477 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,477 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:37,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:37,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:37,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:37,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:37,478 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:37,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bb37078[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:37,478 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,480 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:37,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:37,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:37,480 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,480 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:37,481 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:37,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:37,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:37,482 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:37,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:37,484 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:37,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:37,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:37,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:37,484 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:37,485 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,485 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:37,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:37,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,485 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:37,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:37,486 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:37,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:37,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:37,487 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:37,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:37,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567797480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:37,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567797480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,489 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:37,490 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:37,490 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,490 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,490 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,490 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,490 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,490 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,490 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:37,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:37,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:37,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:37,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:37,490 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:37,490 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:37,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:37,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567797480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:37,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567797481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:37,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567797487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,498 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:37,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:37,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:37,590 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:37,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:37,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:37,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:37,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:37,593 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:37,593 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:37,593 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:37,593 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:37,594 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,594 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,595 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,595 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:37,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:37,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:37,595 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,595 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:37,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:37,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:37,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:37,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:37,596 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:37,596 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:37,596 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:37,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:37,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:37,597 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:37,597 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:37,597 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:37,597 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:37,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:37,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:37,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:37,597 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:37,599 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,599 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:37,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:37,600 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,600 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:37,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:37,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:37,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:37,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:37,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21202c1f[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:37,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:37,601 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,602 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,603 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:37,603 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:37,603 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:37,603 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,603 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:37,607 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:37,607 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,607 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:37,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:37,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:37,607 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:37,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:37,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:37,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:37,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:37,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:37,609 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:37,609 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:37,609 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:37,609 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:37,609 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:37,611 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,611 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:37,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:37,611 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:37,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,611 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:37,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:37,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:37,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:37,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,612 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:37,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:37,616 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:37,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:37,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:37,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:37,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:37,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:37,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:37,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,616 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:37,616 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:37,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:37,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:37,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,629 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=55 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/720fe756ee954d098efe76ac578e6c10 2023-04-27 03:55:37,716 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:37,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:37,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:37,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:37,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:37,719 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:37,719 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:37,719 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:37,719 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:37,720 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,720 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,721 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:37,721 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:37,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:37,721 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,721 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:37,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:37,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:37,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:37,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:37,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:37,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:37,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:37,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:37,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:37,723 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:37,723 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:37,723 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:37,723 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:37,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:37,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:37,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:37,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:37,725 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,725 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:37,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:37,726 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,726 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:37,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:37,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:37,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:37,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:37,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@379e3a25[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:37,728 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:37,728 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,729 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:37,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:37,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:37,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:37,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:37,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:37,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:37,731 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:37,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:37,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:37,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:37,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:37,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:37,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:37,736 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:37,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:37,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:37,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:37,736 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:37,738 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:37,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:37,738 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,738 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:37,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:37,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:37,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:37,739 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:37,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,740 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:37,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,751 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,751 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,751 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:37,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:37,751 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:37,751 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:37,751 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,751 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,751 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:37,751 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:37,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:37,751 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:37,751 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:37,752 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:37,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:37,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:37,852 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:37,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:37,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:37,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:37,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:37,859 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:37,859 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:37,859 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:37,859 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:37,860 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,860 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,861 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:37,861 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:37,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:37,861 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,862 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:37,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:37,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:37,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:37,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:37,863 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:37,863 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:37,863 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:37,863 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:37,863 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:37,865 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:37,865 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:37,865 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:37,865 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:37,865 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:37,865 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:37,865 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,865 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:37,865 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:37,867 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:37,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:37,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:37,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:37,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,869 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,869 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,869 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:37,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:37,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:37,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fe57b14[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:37,870 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:37,870 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,871 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,872 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:37,872 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:37,872 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:37,872 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,872 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:37,875 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:37,875 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:37,875 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:37,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:37,875 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:37,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:37,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:37,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:37,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:37,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:37,877 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:37,877 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:37,877 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:37,877 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:37,877 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:37,879 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,879 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:37,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:37,879 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:37,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:37,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:37,880 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:37,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:37,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,880 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:37,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,884 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,884 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,884 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,884 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,884 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,884 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,884 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,884 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:37,884 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:37,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:37,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:37,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:37,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:37,884 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:37,885 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:37,885 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:37,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:37,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:37,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:37,985 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:37,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:37,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:37,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:37,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:37,988 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:37,989 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:37,989 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:37,989 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:37,989 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,989 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,990 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:37,990 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:37,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:37,991 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,991 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:37,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:37,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:37,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:37,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:37,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:37,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:37,992 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:37,992 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:37,992 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:37,993 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:37,993 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:37,993 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:37,993 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:37,994 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:37,994 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:37,994 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:37,994 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:37,994 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,994 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:37,994 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:37,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,996 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:37,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:37,997 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,997 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:37,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:37,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:37,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:37,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:37,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:37,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:37,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f65e620[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:37,998 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:37,998 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,999 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:37,999 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:37,999 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:37,999 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:37,999 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,000 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:38,001 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:38,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:38,001 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,001 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:38,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:38,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:38,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567797991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,001 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:38,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:38,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:38,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567797992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:38,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:38,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567797993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:38,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:38,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567797995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:38,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567797999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:38,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:38,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:38,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:38,009 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:38,009 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:38,009 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:38,009 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:38,009 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:38,010 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,010 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,010 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:38,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:38,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:38,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:38,011 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:38,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:38,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:38,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:38,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:38,012 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:38,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:38,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,022 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,022 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:38,022 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:38,022 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,022 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:38,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:38,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:38,022 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:38,023 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:38,022 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,023 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,023 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:38,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:38,023 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:38,023 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:38,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:38,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:38,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:38,038 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=55 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/99cc2c75820c4a36af3ddf66a9f11725 2023-04-27 03:55:38,043 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/8fb17fae54ab4d22892cf2642bba8a19 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/8fb17fae54ab4d22892cf2642bba8a19 2023-04-27 03:55:38,048 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/8fb17fae54ab4d22892cf2642bba8a19, entries=300, sequenceid=55, filesize=18.5 K 2023-04-27 03:55:38,049 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/720fe756ee954d098efe76ac578e6c10 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/720fe756ee954d098efe76ac578e6c10 2023-04-27 03:55:38,052 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/720fe756ee954d098efe76ac578e6c10, entries=150, sequenceid=55, filesize=11.6 K 2023-04-27 03:55:38,053 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/99cc2c75820c4a36af3ddf66a9f11725 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/99cc2c75820c4a36af3ddf66a9f11725 2023-04-27 03:55:38,055 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/99cc2c75820c4a36af3ddf66a9f11725, entries=150, sequenceid=55, filesize=11.6 K 2023-04-27 03:55:38,056 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 98cbc3f1a78abce9e99d0349272cd6a4 in 1246ms, sequenceid=55, compaction requested=true 2023-04-27 03:55:38,056 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:38,056 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:38,056 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:38,056 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:38,056 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:38,056 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:38,057 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42678 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:38,057 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/A is initiating minor compaction (all files) 2023-04-27 03:55:38,057 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/A in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:38,057 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/340996cbeed044d7967aafbf4bd8b0b1, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/e9a8dc6e26f646e788f6576b56d628ea, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/8fb17fae54ab4d22892cf2642bba8a19] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=41.7 K 2023-04-27 03:55:38,057 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:38,058 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/B is initiating minor compaction (all files) 2023-04-27 03:55:38,058 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/B in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:38,058 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/5f717233d77c49cd908915e38a25776d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/0acae6104d8645cbb87c450b5608dc1f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/720fe756ee954d098efe76ac578e6c10] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=34.8 K 2023-04-27 03:55:38,058 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 340996cbeed044d7967aafbf4bd8b0b1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682567734644 2023-04-27 03:55:38,059 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting e9a8dc6e26f646e788f6576b56d628ea, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=44, earliestPutTs=1682567734691 2023-04-27 03:55:38,059 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 5f717233d77c49cd908915e38a25776d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682567734644 2023-04-27 03:55:38,060 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 0acae6104d8645cbb87c450b5608dc1f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=44, earliestPutTs=1682567734691 2023-04-27 03:55:38,060 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 8fb17fae54ab4d22892cf2642bba8a19, keycount=300, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=55, earliestPutTs=1682567736798 2023-04-27 03:55:38,060 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 720fe756ee954d098efe76ac578e6c10, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=55, earliestPutTs=1682567736798 2023-04-27 03:55:38,066 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#A#compaction#407 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:38,068 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#B#compaction#408 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:38,123 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:38,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:38,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:38,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:38,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:38,129 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:38,129 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:38,129 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:38,129 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:38,129 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,130 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:38,131 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:38,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:38,131 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:38,131 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,131 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:38,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:38,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:38,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:38,132 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:38,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:38,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:38,132 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:38,132 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:38,132 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:38,134 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:38,134 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:38,134 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:38,134 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:38,134 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-27 03:55:38,134 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:38,134 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:38,134 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:38,134 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:38,134 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:38,134 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:38,141 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/c71245de8e0d44209eaa10ef6ec67eeb 2023-04-27 03:55:38,164 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/c0fd3b61109d4c6c8d56fb9b626a92b0 2023-04-27 03:55:38,486 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/dff975d2a9c94275a4d49bb96f65916c as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/dff975d2a9c94275a4d49bb96f65916c 2023-04-27 03:55:38,489 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/8b4e040d4f0c47839c0e80d945ffbe75 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/8b4e040d4f0c47839c0e80d945ffbe75 2023-04-27 03:55:38,490 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/A of 98cbc3f1a78abce9e99d0349272cd6a4 into dff975d2a9c94275a4d49bb96f65916c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:38,490 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:38,490 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/A, priority=13, startTime=1682567738056; duration=0sec 2023-04-27 03:55:38,490 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:38,490 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:38,491 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:38,491 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/C is initiating minor compaction (all files) 2023-04-27 03:55:38,491 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/C in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:38,492 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/402f32b9f7cd4fcb970cf24d47e9cf1a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/c386f0ca13eb4b669ad7738398aa9922, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/99cc2c75820c4a36af3ddf66a9f11725] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=34.8 K 2023-04-27 03:55:38,492 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 402f32b9f7cd4fcb970cf24d47e9cf1a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682567734644 2023-04-27 03:55:38,492 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting c386f0ca13eb4b669ad7738398aa9922, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=44, earliestPutTs=1682567734691 2023-04-27 03:55:38,493 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 99cc2c75820c4a36af3ddf66a9f11725, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=55, earliestPutTs=1682567736798 2023-04-27 03:55:38,493 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/B of 98cbc3f1a78abce9e99d0349272cd6a4 into 8b4e040d4f0c47839c0e80d945ffbe75(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:38,493 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:38,493 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/B, priority=13, startTime=1682567738056; duration=0sec 2023-04-27 03:55:38,493 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:38,498 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#C#compaction#412 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:38,594 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/3528149d7af945e9988b065a88bf6ee0 2023-04-27 03:55:38,598 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/c71245de8e0d44209eaa10ef6ec67eeb as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/c71245de8e0d44209eaa10ef6ec67eeb 2023-04-27 03:55:38,601 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/c71245de8e0d44209eaa10ef6ec67eeb, entries=150, sequenceid=91, filesize=11.6 K 2023-04-27 03:55:38,601 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/c0fd3b61109d4c6c8d56fb9b626a92b0 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/c0fd3b61109d4c6c8d56fb9b626a92b0 2023-04-27 03:55:38,606 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/c0fd3b61109d4c6c8d56fb9b626a92b0, entries=150, sequenceid=91, filesize=11.6 K 2023-04-27 03:55:38,607 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/3528149d7af945e9988b065a88bf6ee0 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/3528149d7af945e9988b065a88bf6ee0 2023-04-27 03:55:38,609 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/3528149d7af945e9988b065a88bf6ee0, entries=150, sequenceid=91, filesize=11.6 K 2023-04-27 03:55:38,610 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=0 B/0 for 98cbc3f1a78abce9e99d0349272cd6a4 in 476ms, sequenceid=91, compaction requested=false 2023-04-27 03:55:38,610 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:38,610 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:38,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:38,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:38,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:38,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:38,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,613 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:38,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:38,614 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,614 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:38,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:38,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:38,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:38,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:38,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:38,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f057907[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:38,615 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:38,615 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,617 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:38,617 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:38,617 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:38,617 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,617 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:38,619 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:38,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:38,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:38,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:38,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:38,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:38,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:38,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:38,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:38,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:38,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:38,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:38,622 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:38,622 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:38,622 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:38,622 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:38,622 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:38,624 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,624 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,624 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:38,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:38,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:38,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:38,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:38,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:38,625 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:38,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:38,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:38,626 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:38,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:38,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:38,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:38,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,631 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:38,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:38,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:38,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:38,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:38,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:38,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:38,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,631 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:38,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:38,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:38,631 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:38,731 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:38,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:38,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:38,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:38,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:38,737 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:38,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:38,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:38,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:38,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:38,738 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:38,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:38,739 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:38,739 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,739 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:38,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:38,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:38,740 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:38,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:38,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:38,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:38,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:38,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:38,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:38,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:38,760 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:38,760 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:38,761 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:38,761 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:38,761 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:38,761 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:38,761 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,761 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:38,761 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:38,763 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:38,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:38,764 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:38,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:38,764 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,764 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:38,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:38,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:38,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:38,765 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:38,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@390c2bd0[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:38,765 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,767 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,767 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:38,767 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:38,767 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:38,767 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,767 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:38,769 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,769 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:38,769 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:38,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:38,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:38,769 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:38,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:38,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:38,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:38,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:38,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:38,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:38,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:38,771 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:38,771 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:38,771 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:38,771 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:38,771 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:38,773 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,773 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:38,773 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:38,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:38,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:38,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:38,773 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:38,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:38,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:38,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:38,774 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:38,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:38,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,777 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,777 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:38,777 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:38,777 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,777 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:38,778 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:38,778 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:38,777 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,778 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:38,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:38,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:38,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:38,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:38,778 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,778 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:38,778 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:38,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:38,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:38,878 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:38,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:38,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:38,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:38,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:38,882 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:38,882 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:38,882 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:38,882 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:38,882 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,882 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:38,884 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:38,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:38,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:38,884 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:38,884 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,885 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:38,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:38,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:38,885 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:38,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:38,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:38,888 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:38,888 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:38,888 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:38,892 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:38,893 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:38,895 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:38,895 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:38,895 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:38,895 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:38,895 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:38,895 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:38,895 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,895 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:38,895 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:38,897 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,897 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:38,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:38,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:38,898 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,898 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:38,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:38,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:38,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:38,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:38,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7df5ca3e[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:38,899 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:38,899 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,901 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,901 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:38,901 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:38,901 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:38,901 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,901 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:38,904 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:38,904 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,904 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:38,904 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:38,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:38,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:38,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:38,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:38,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:38,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:38,907 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/d1653a0473b94e4fae74bdd411b3e153 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/d1653a0473b94e4fae74bdd411b3e153 2023-04-27 03:55:38,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:38,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:38,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:38,908 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:38,908 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:38,909 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:38,909 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:38,909 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:38,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:38,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:38,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:38,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:38,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:38,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:38,912 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/C of 98cbc3f1a78abce9e99d0349272cd6a4 into d1653a0473b94e4fae74bdd411b3e153(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:38,912 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:38,912 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/C, priority=13, startTime=1682567738056; duration=0sec 2023-04-27 03:55:38,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,912 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:38,913 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:38,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:38,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:38,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,914 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:38,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:38,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,921 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:38,921 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:38,921 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:38,921 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:38,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:38,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:38,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:38,921 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:38,922 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:38,922 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:38,922 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,922 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:38,922 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:38,922 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:38,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:38,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:38,922 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:39,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:39,014 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:55:39,015 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:39,015 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:39,015 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:39,015 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:39,015 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:39,015 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:39,023 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:39,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:39,025 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/5a56b48fc6ea41b8877bae4d8d1226ad 2023-04-27 03:55:39,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:39,037 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/6d7ac12827f84d4497b9791cfb3bc138 2023-04-27 03:55:39,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:39,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:39,042 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:39,042 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:39,042 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:39,042 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:39,044 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,044 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,045 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:39,045 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:39,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:39,046 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,046 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:39,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:39,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:39,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:39,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:39,048 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:39,048 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:39,048 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:39,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567739042, max timestamp=1682567739046Num uniques 859; 2023-04-27 03:55:39,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567739042, max timestamp=1682567739046Num uniques 859; 2023-04-27 03:55:39,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:55:39,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:55:39,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567739042, max timestamp=1682567739046Num uniques 859; 2023-04-27 03:55:39,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:55:39,059 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:39,059 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:39,059 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:39,059 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:39,059 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:39,059 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:39,059 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:39,059 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:39,059 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,060 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:39,060 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:39,062 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,062 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:39,062 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,062 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:39,062 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/1a05f92b3b1b4bfa96d7e4666200ed06 2023-04-27 03:55:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:39,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:39,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dd16746[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:39,064 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:39,064 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,069 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/5a56b48fc6ea41b8877bae4d8d1226ad as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/5a56b48fc6ea41b8877bae4d8d1226ad 2023-04-27 03:55:39,070 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,070 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:39,070 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:39,070 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:39,072 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:39,072 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:39,072 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:39,073 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,073 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:39,073 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:39,073 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:39,073 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:39,074 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:39,074 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:39,074 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:39,074 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/5a56b48fc6ea41b8877bae4d8d1226ad, entries=200, sequenceid=106, filesize=13.9 K 2023-04-27 03:55:39,075 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/6d7ac12827f84d4497b9791cfb3bc138 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/6d7ac12827f84d4497b9791cfb3bc138 2023-04-27 03:55:39,077 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:39,078 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,078 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:39,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:39,078 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,079 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/6d7ac12827f84d4497b9791cfb3bc138, entries=150, sequenceid=106, filesize=11.6 K 2023-04-27 03:55:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,079 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/1a05f92b3b1b4bfa96d7e4666200ed06 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/1a05f92b3b1b4bfa96d7e4666200ed06 2023-04-27 03:55:39,080 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:39,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,082 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:39,082 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:39,082 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:39,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:39,082 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:39,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:39,082 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:39,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:39,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:39,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:39,082 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,083 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,083 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,083 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,083 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,083 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:39,083 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:39,083 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/1a05f92b3b1b4bfa96d7e4666200ed06, entries=150, sequenceid=106, filesize=11.6 K 2023-04-27 03:55:39,084 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=114.05 KB/116790 for 98cbc3f1a78abce9e99d0349272cd6a4 in 70ms, sequenceid=106, compaction requested=true 2023-04-27 03:55:39,084 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:39,084 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:39,084 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:39,084 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:39,084 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:39,084 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:39,085 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:39,085 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/B is initiating minor compaction (all files) 2023-04-27 03:55:39,085 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38101 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:39,085 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/B in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:39,085 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/A is initiating minor compaction (all files) 2023-04-27 03:55:39,085 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/8b4e040d4f0c47839c0e80d945ffbe75, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/c0fd3b61109d4c6c8d56fb9b626a92b0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/6d7ac12827f84d4497b9791cfb3bc138] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=34.9 K 2023-04-27 03:55:39,085 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/A in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:39,085 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/dff975d2a9c94275a4d49bb96f65916c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/c71245de8e0d44209eaa10ef6ec67eeb, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/5a56b48fc6ea41b8877bae4d8d1226ad] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=37.2 K 2023-04-27 03:55:39,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:39,086 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-27 03:55:39,086 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 8b4e040d4f0c47839c0e80d945ffbe75, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=55, earliestPutTs=1682567736798 2023-04-27 03:55:39,086 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting dff975d2a9c94275a4d49bb96f65916c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=55, earliestPutTs=1682567736798 2023-04-27 03:55:39,087 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting c0fd3b61109d4c6c8d56fb9b626a92b0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1682567736862 2023-04-27 03:55:39,087 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting c71245de8e0d44209eaa10ef6ec67eeb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1682567736862 2023-04-27 03:55:39,087 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 6d7ac12827f84d4497b9791cfb3bc138, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1682567739008 2023-04-27 03:55:39,087 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:39,087 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 5a56b48fc6ea41b8877bae4d8d1226ad, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1682567739008 2023-04-27 03:55:39,087 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:39,087 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:39,087 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:39,088 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:39,088 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:39,101 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#B#compaction#417 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:39,102 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#A#compaction#418 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:39,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567799101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567799103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567799102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567799112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567799116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,183 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:39,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:39,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:39,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:39,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:39,187 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:39,187 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:39,187 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:39,187 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:39,187 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,187 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,189 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,189 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:39,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:39,190 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,190 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:39,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:39,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:39,192 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:39,192 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:39,192 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:39,192 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:39,192 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:39,192 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:39,192 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:39,192 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:39,192 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:39,193 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:39,193 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:39,193 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,193 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:39,193 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:39,194 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,194 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:39,195 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,195 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:39,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:39,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5948faba[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:39,196 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:39,196 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,197 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,198 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:39,198 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:39,198 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:39,198 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,198 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:39,199 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:39,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:39,199 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:39,199 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:39,199 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:39,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:39,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:39,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:39,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:39,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:39,201 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:39,201 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:39,201 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:39,201 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:39,201 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:39,204 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,204 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:39,204 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:39,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:39,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:39,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:39,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:39,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,205 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:39,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,206 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:39,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,208 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,208 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,208 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:39,208 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:39,208 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:39,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:39,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:39,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:39,208 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,209 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,209 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,208 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:39,209 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:39,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:39,209 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:39,209 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:39,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567799213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567799213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567799217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567799227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567799227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,309 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:39,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:39,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:39,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:39,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:39,312 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:39,312 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:39,313 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:39,313 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:39,313 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,313 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,314 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:39,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,315 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:39,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:39,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:39,316 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:39,316 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:39,316 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:39,316 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:39,316 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:39,316 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:39,316 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:39,316 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:39,316 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:39,316 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:39,316 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:39,316 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,316 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:39,316 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:39,317 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,318 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,318 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,318 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:39,318 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35aa4db9[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:39,318 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:39,318 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:39,318 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,318 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,318 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:39,320 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,320 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:39,320 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:39,320 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:39,320 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,320 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:39,322 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,322 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:39,322 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:39,322 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:39,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:39,324 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:39,324 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:39,324 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:39,324 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:39,324 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:39,326 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,326 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,326 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:39,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:39,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:39,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:39,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:39,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:39,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:39,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,328 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:39,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,329 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,331 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:39,331 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:39,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:39,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:39,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:39,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:39,331 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:39,332 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,332 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,332 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,332 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,332 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,332 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,332 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,332 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:39,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:39,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:39,332 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:39,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567799418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567799420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,432 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:39,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:39,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:39,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567799429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,435 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567799431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567799431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:39,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:39,441 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:39,441 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:39,441 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:39,441 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:39,441 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,441 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,447 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:39,447 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,447 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,449 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:39,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:39,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:39,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:39,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:39,450 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:39,450 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:39,450 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:39,450 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:39,450 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:39,450 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:39,450 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:39,450 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:39,451 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:39,451 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:39,451 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:39,451 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,451 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:39,451 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:39,452 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:39,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:39,452 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:39,454 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,454 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:39,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:39,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:39,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:39,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14218e8[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:39,456 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:39,456 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,457 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,457 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:39,457 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:39,457 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:39,457 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,457 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:39,459 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,459 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:39,459 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:39,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,459 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:39,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:39,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:39,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:39,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:39,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:39,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:39,465 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:39,465 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:39,465 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:39,465 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:39,465 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:39,467 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,467 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:39,467 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:39,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:39,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:39,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:39,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:39,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,470 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:39,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,472 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:39,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,481 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:39,481 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:39,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:39,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:39,481 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,481 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,481 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,481 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,481 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,481 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,481 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:39,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:39,481 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:39,481 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:39,481 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:39,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:39,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:39,515 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/be3bb316bfed451f9839df444860fa6c 2023-04-27 03:55:39,523 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/77dbc1cc0fb649c3811c1488a1fd86d6 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/77dbc1cc0fb649c3811c1488a1fd86d6 2023-04-27 03:55:39,531 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/B of 98cbc3f1a78abce9e99d0349272cd6a4 into 77dbc1cc0fb649c3811c1488a1fd86d6(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:39,532 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:39,532 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/B, priority=13, startTime=1682567739084; duration=0sec 2023-04-27 03:55:39,532 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:39,532 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:39,533 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:39,533 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/C is initiating minor compaction (all files) 2023-04-27 03:55:39,533 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/C in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:39,533 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/d1653a0473b94e4fae74bdd411b3e153, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/3528149d7af945e9988b065a88bf6ee0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/1a05f92b3b1b4bfa96d7e4666200ed06] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=34.9 K 2023-04-27 03:55:39,534 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/b798b1fda6cb4ab68f09c6f650d337ac as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/b798b1fda6cb4ab68f09c6f650d337ac 2023-04-27 03:55:39,534 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting d1653a0473b94e4fae74bdd411b3e153, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=55, earliestPutTs=1682567736798 2023-04-27 03:55:39,534 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 3528149d7af945e9988b065a88bf6ee0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1682567736862 2023-04-27 03:55:39,535 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 1a05f92b3b1b4bfa96d7e4666200ed06, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1682567739008 2023-04-27 03:55:39,540 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/A of 98cbc3f1a78abce9e99d0349272cd6a4 into b798b1fda6cb4ab68f09c6f650d337ac(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:39,540 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:39,540 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/A, priority=13, startTime=1682567739084; duration=0sec 2023-04-27 03:55:39,540 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:39,547 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#C#compaction#420 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:39,548 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/21a6756e47074d6dbb50430a1b1a8c7c 2023-04-27 03:55:39,566 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/dc21c6e1598a4308ad28470f893ebf9c as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/dc21c6e1598a4308ad28470f893ebf9c 2023-04-27 03:55:39,569 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/C of 98cbc3f1a78abce9e99d0349272cd6a4 into dc21c6e1598a4308ad28470f893ebf9c(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:39,569 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:39,569 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/C, priority=13, startTime=1682567739084; duration=0sec 2023-04-27 03:55:39,570 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:39,577 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/c681f8685e7d455a9f61ddfb9451531e 2023-04-27 03:55:39,581 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/be3bb316bfed451f9839df444860fa6c as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/be3bb316bfed451f9839df444860fa6c 2023-04-27 03:55:39,582 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:39,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:39,585 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/be3bb316bfed451f9839df444860fa6c, entries=250, sequenceid=141, filesize=16.4 K 2023-04-27 03:55:39,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:39,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:39,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:39,586 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:39,587 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:39,587 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:39,587 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:39,587 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,587 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,588 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,588 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:39,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:39,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:39,588 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,588 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:39,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:39,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:39,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:39,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:39,589 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:39,589 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:39,589 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:39,592 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:39,592 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:39,592 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:39,592 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:39,592 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:39,592 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:39,592 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:39,592 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:39,592 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,592 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:39,592 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:39,593 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,594 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:39,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:39,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:39,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:39,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,595 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,595 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:39,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,599 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/21a6756e47074d6dbb50430a1b1a8c7c as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/21a6756e47074d6dbb50430a1b1a8c7c 2023-04-27 03:55:39,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:39,604 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/21a6756e47074d6dbb50430a1b1a8c7c, entries=150, sequenceid=141, filesize=11.8 K 2023-04-27 03:55:39,604 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/c681f8685e7d455a9f61ddfb9451531e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/c681f8685e7d455a9f61ddfb9451531e 2023-04-27 03:55:39,607 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/c681f8685e7d455a9f61ddfb9451531e, entries=150, sequenceid=141, filesize=11.8 K 2023-04-27 03:55:39,608 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=60.38 KB/61830 for 98cbc3f1a78abce9e99d0349272cd6a4 in 522ms, sequenceid=141, compaction requested=false 2023-04-27 03:55:39,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:39,608 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:39,608 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:39,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52b596c4[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:39,608 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,610 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,611 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:39,611 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:39,611 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,611 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:39,611 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:39,612 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:39,612 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:39,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,612 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:39,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:39,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:39,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:39,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:39,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:39,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:39,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:39,616 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:39,616 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:39,616 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:39,616 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:39,616 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:39,617 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,617 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,617 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:39,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:39,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:39,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:39,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:39,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:39,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,618 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,620 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:39,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:39,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:39,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:39,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:39,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:39,623 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:39,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:39,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:39,623 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:39,623 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:39,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:39,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:39,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:39,723 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:39,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:39,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:39,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:39,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:39,727 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:39,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:39,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:39,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:39,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,728 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:39,729 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-27 03:55:39,729 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:39,729 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:39,729 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:39,729 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:39,729 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:39,729 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:39,730 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:39,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:39,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:39,730 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:39,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:39,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:39,732 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:39,732 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:39,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:39,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:39,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:39,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:39,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:39,732 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:39,733 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:39,733 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:39,733 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:39,733 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:39,733 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:39,733 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,733 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:39,733 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:39,734 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,734 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:39,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:39,735 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,735 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:39,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:39,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:39,739 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/a63962a56ca74007b15dddfa151d74c9 2023-04-27 03:55:39,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:39,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:39,746 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:39,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54583083[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:39,746 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,747 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,748 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:39,748 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:39,748 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,748 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:39,748 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:39,749 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:39,749 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:39,749 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:39,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:39,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:39,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:39,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:39,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:39,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:39,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:39,752 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:39,752 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:39,752 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:39,752 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:39,752 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:39,753 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/86d15e64f2354e249a8622e41fe51fb9 2023-04-27 03:55:39,755 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,755 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:39,755 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:39,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:39,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:39,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:39,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,756 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:39,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:39,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,757 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:39,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,760 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:39,760 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:39,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:39,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:39,761 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:39,761 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,761 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,761 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,761 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:39,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:39,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:39,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:39,761 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:39,761 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,761 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,761 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:39,761 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:39,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567799756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567799756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,769 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567799757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,769 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567799759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,769 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567799765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,795 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/340996cbeed044d7967aafbf4bd8b0b1, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/e9a8dc6e26f646e788f6576b56d628ea, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/8fb17fae54ab4d22892cf2642bba8a19, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/dff975d2a9c94275a4d49bb96f65916c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/c71245de8e0d44209eaa10ef6ec67eeb, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/5a56b48fc6ea41b8877bae4d8d1226ad] to archive 2023-04-27 03:55:39,796 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 03:55:39,800 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/340996cbeed044d7967aafbf4bd8b0b1 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/340996cbeed044d7967aafbf4bd8b0b1 2023-04-27 03:55:39,801 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/e9a8dc6e26f646e788f6576b56d628ea to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/e9a8dc6e26f646e788f6576b56d628ea 2023-04-27 03:55:39,804 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/8fb17fae54ab4d22892cf2642bba8a19 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/8fb17fae54ab4d22892cf2642bba8a19 2023-04-27 03:55:39,805 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/dff975d2a9c94275a4d49bb96f65916c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/dff975d2a9c94275a4d49bb96f65916c 2023-04-27 03:55:39,806 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/c71245de8e0d44209eaa10ef6ec67eeb to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/c71245de8e0d44209eaa10ef6ec67eeb 2023-04-27 03:55:39,807 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/5a56b48fc6ea41b8877bae4d8d1226ad to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/5a56b48fc6ea41b8877bae4d8d1226ad 2023-04-27 03:55:39,824 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/5f717233d77c49cd908915e38a25776d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/0acae6104d8645cbb87c450b5608dc1f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/8b4e040d4f0c47839c0e80d945ffbe75, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/720fe756ee954d098efe76ac578e6c10, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/c0fd3b61109d4c6c8d56fb9b626a92b0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/6d7ac12827f84d4497b9791cfb3bc138] to archive 2023-04-27 03:55:39,825 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 03:55:39,826 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/5f717233d77c49cd908915e38a25776d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/5f717233d77c49cd908915e38a25776d 2023-04-27 03:55:39,827 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/0acae6104d8645cbb87c450b5608dc1f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/0acae6104d8645cbb87c450b5608dc1f 2023-04-27 03:55:39,828 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/8b4e040d4f0c47839c0e80d945ffbe75 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/8b4e040d4f0c47839c0e80d945ffbe75 2023-04-27 03:55:39,830 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/720fe756ee954d098efe76ac578e6c10 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/720fe756ee954d098efe76ac578e6c10 2023-04-27 03:55:39,831 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/c0fd3b61109d4c6c8d56fb9b626a92b0 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/c0fd3b61109d4c6c8d56fb9b626a92b0 2023-04-27 03:55:39,832 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/6d7ac12827f84d4497b9791cfb3bc138 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/6d7ac12827f84d4497b9791cfb3bc138 2023-04-27 03:55:39,847 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/402f32b9f7cd4fcb970cf24d47e9cf1a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/c386f0ca13eb4b669ad7738398aa9922, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/d1653a0473b94e4fae74bdd411b3e153, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/99cc2c75820c4a36af3ddf66a9f11725, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/3528149d7af945e9988b065a88bf6ee0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/1a05f92b3b1b4bfa96d7e4666200ed06] to archive 2023-04-27 03:55:39,847 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 03:55:39,849 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/402f32b9f7cd4fcb970cf24d47e9cf1a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/402f32b9f7cd4fcb970cf24d47e9cf1a 2023-04-27 03:55:39,851 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/c386f0ca13eb4b669ad7738398aa9922 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/c386f0ca13eb4b669ad7738398aa9922 2023-04-27 03:55:39,852 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/d1653a0473b94e4fae74bdd411b3e153 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/d1653a0473b94e4fae74bdd411b3e153 2023-04-27 03:55:39,853 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/99cc2c75820c4a36af3ddf66a9f11725 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/99cc2c75820c4a36af3ddf66a9f11725 2023-04-27 03:55:39,854 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/3528149d7af945e9988b065a88bf6ee0 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/3528149d7af945e9988b065a88bf6ee0 2023-04-27 03:55:39,855 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/1a05f92b3b1b4bfa96d7e4666200ed06 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/1a05f92b3b1b4bfa96d7e4666200ed06 2023-04-27 03:55:39,861 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:39,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:39,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:39,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:39,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:39,865 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:39,865 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:39,865 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:39,865 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:39,866 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,866 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,867 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:39,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:39,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:39,867 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,868 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,868 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:39,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:39,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,869 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:39,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567799866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567799866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,872 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:39,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:39,872 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:39,872 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:39,872 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:39,876 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:39,876 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:39,876 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:39,876 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:39,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,876 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:39,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567799870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,876 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:39,876 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:39,876 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:39,876 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,877 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:39,877 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:39,878 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,878 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,879 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:39,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:39,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567799871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567799871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,882 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:39,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:39,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:39,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:39,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:39,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:39,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a5b9d98[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:39,884 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:39,884 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,890 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,890 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:39,890 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:39,890 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:39,891 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,891 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:39,891 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,891 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:39,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:39,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:39,891 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:39,892 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:39,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:39,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:39,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:39,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:39,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:39,894 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:39,894 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:39,894 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:39,894 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:39,894 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:39,896 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,896 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:39,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:39,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,896 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:39,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:39,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:39,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:39,896 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:39,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,897 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:39,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:39,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:39,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,900 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:39,900 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:39,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:39,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:39,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:39,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:39,901 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,901 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,901 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:39,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:39,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:39,901 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:39,901 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,901 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:39,901 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:39,901 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:39,901 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:39,901 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,001 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:40,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:40,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:40,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:40,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:40,004 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:40,005 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:40,005 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:40,005 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:40,005 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,005 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:40,008 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:40,008 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:40,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:40,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:40,008 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,008 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:40,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:40,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:40,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:40,009 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:40,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:40,009 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:40,009 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:40,009 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:40,009 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:40,009 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:40,009 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:40,009 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:40,009 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:40,009 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:40,009 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:40,009 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:40,009 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,009 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:40,009 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:40,011 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,011 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:40,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:40,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:40,011 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:40,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:40,011 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:40,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:40,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:40,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:40,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:40,013 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:40,013 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:40,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b4f4963[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:40,014 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:40,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:40,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:40,014 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:40,014 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,014 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:40,014 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:40,014 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:40,015 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,015 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:40,015 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:40,016 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:40,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:40,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:40,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:40,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:40,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:40,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:40,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:40,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:40,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:40,018 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:40,018 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:40,018 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:40,018 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:40,018 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:40,023 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,023 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:40,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:40,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,023 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:40,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:40,024 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:40,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:40,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:40,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:40,025 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:40,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,029 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,029 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:40,029 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:40,029 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,029 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:40,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:40,029 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:40,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:40,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:40,029 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:40,029 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:40,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:40,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:40,029 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:40,029 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,029 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:40,030 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:40,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:40,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:40,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:40,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:40,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567800071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:40,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567800072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:40,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567800077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:40,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567800081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:40,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567800082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,130 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:40,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:40,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:40,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:40,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:40,136 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:40,136 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:40,136 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:40,136 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:40,136 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,137 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:40,138 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,138 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:40,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:40,138 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,138 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:40,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:40,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:40,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:40,155 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:40,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:40,156 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:40,156 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:40,156 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:40,161 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:40,161 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:40,162 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:40,162 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:40,162 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:40,162 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:40,162 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:40,162 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:40,162 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,162 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:40,162 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:40,164 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/8e691d70025f40718b9d060095c96d91 2023-04-27 03:55:40,165 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:40,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:40,165 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:40,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:40,166 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:40,166 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:40,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:40,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:40,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:40,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:40,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2532e778[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:40,168 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:40,168 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:40,169 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/a63962a56ca74007b15dddfa151d74c9 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/a63962a56ca74007b15dddfa151d74c9 2023-04-27 03:55:40,171 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:40,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:40,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:40,172 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:40,172 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:40,172 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:40,172 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,172 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:40,175 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/a63962a56ca74007b15dddfa151d74c9, entries=150, sequenceid=157, filesize=11.8 K 2023-04-27 03:55:40,175 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:40,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:40,175 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:40,175 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/86d15e64f2354e249a8622e41fe51fb9 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/86d15e64f2354e249a8622e41fe51fb9 2023-04-27 03:55:40,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:40,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:40,175 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:40,176 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:40,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:40,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:40,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:40,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:40,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:40,178 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:40,179 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:40,179 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:40,179 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:40,179 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:40,180 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/86d15e64f2354e249a8622e41fe51fb9, entries=150, sequenceid=157, filesize=11.8 K 2023-04-27 03:55:40,180 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,180 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:40,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,180 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:40,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:40,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:40,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:40,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:40,182 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,182 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/8e691d70025f40718b9d060095c96d91 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/8e691d70025f40718b9d060095c96d91 2023-04-27 03:55:40,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:40,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:40,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:40,184 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:40,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:40,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,187 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:40,188 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:40,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:40,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:40,189 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,189 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:40,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:40,189 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:40,189 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:40,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:40,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:40,189 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,189 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:40,189 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:40,189 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:40,190 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:40,190 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:40,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:40,193 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/8e691d70025f40718b9d060095c96d91, entries=150, sequenceid=157, filesize=11.8 K 2023-04-27 03:55:40,194 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 98cbc3f1a78abce9e99d0349272cd6a4 in 465ms, sequenceid=157, compaction requested=true 2023-04-27 03:55:40,194 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:40,195 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:40,195 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:40,195 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:40,195 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:40,195 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:40,196 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40944 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:40,196 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/A is initiating minor compaction (all files) 2023-04-27 03:55:40,196 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/A in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:40,196 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/b798b1fda6cb4ab68f09c6f650d337ac, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/be3bb316bfed451f9839df444860fa6c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/a63962a56ca74007b15dddfa151d74c9] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=40.0 K 2023-04-27 03:55:40,196 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36164 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:40,196 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/B is initiating minor compaction (all files) 2023-04-27 03:55:40,196 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/B in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:40,196 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/77dbc1cc0fb649c3811c1488a1fd86d6, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/21a6756e47074d6dbb50430a1b1a8c7c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/86d15e64f2354e249a8622e41fe51fb9] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=35.3 K 2023-04-27 03:55:40,197 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting b798b1fda6cb4ab68f09c6f650d337ac, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1682567739008 2023-04-27 03:55:40,197 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 77dbc1cc0fb649c3811c1488a1fd86d6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1682567739008 2023-04-27 03:55:40,197 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting be3bb316bfed451f9839df444860fa6c, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1682567739070 2023-04-27 03:55:40,197 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 21a6756e47074d6dbb50430a1b1a8c7c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1682567739074 2023-04-27 03:55:40,198 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 86d15e64f2354e249a8622e41fe51fb9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682567739108 2023-04-27 03:55:40,198 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting a63962a56ca74007b15dddfa151d74c9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682567739108 2023-04-27 03:55:40,205 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#B#compaction#425 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:40,221 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#A#compaction#426 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:40,290 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:40,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:40,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:40,294 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/97ef251156094006bf37bf156e1825b4 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/97ef251156094006bf37bf156e1825b4 2023-04-27 03:55:40,300 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/A of 98cbc3f1a78abce9e99d0349272cd6a4 into 97ef251156094006bf37bf156e1825b4(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:40,300 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:40,300 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/A, priority=13, startTime=1682567740195; duration=0sec 2023-04-27 03:55:40,300 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:40,300 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:40,301 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36164 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:40,301 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/C is initiating minor compaction (all files) 2023-04-27 03:55:40,301 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/C in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:40,301 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/dc21c6e1598a4308ad28470f893ebf9c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/c681f8685e7d455a9f61ddfb9451531e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/8e691d70025f40718b9d060095c96d91] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=35.3 K 2023-04-27 03:55:40,302 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting dc21c6e1598a4308ad28470f893ebf9c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1682567739008 2023-04-27 03:55:40,302 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting c681f8685e7d455a9f61ddfb9451531e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1682567739074 2023-04-27 03:55:40,303 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 8e691d70025f40718b9d060095c96d91, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682567739108 2023-04-27 03:55:40,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:40,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:40,318 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:40,318 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:40,318 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:40,318 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:40,318 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,318 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:40,321 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,321 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:40,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:40,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:40,321 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,321 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:40,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:40,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:40,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:40,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:40,322 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#C#compaction#427 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:40,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:40,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:40,323 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:40,323 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:40,323 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:40,331 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:40,331 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:40,331 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:40,331 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:40,331 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-27 03:55:40,331 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:40,331 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:40,331 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:40,331 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:40,331 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:40,331 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:40,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:40,382 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:40,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:40,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567800399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:40,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567800401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:40,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567800401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:40,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567800403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:40,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567800409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:40,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567800512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:40,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567800514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:40,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567800514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:40,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567800514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:40,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567800515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,696 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/d6b6b1f57c67492d860036d1f5325f94 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/d6b6b1f57c67492d860036d1f5325f94 2023-04-27 03:55:40,701 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/B of 98cbc3f1a78abce9e99d0349272cd6a4 into d6b6b1f57c67492d860036d1f5325f94(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:40,701 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:40,701 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/B, priority=13, startTime=1682567740195; duration=0sec 2023-04-27 03:55:40,701 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:40,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:40,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567800721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:40,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567800725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:40,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567800726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:40,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567800726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:40,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567800728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,770 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/6eb198f1655e429a813e22a3852428ae as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/6eb198f1655e429a813e22a3852428ae 2023-04-27 03:55:40,776 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/C of 98cbc3f1a78abce9e99d0349272cd6a4 into 6eb198f1655e429a813e22a3852428ae(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:40,776 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:40,776 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/C, priority=13, startTime=1682567740195; duration=0sec 2023-04-27 03:55:40,776 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:40,817 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/38eb9f2769e14718900f2b56c6abea59 2023-04-27 03:55:40,857 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/19974a6638f34a6187fad2e600337ff5 2023-04-27 03:55:40,909 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/8a6b439808544a3ebe2377ccdbd47ed2 2023-04-27 03:55:40,913 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/38eb9f2769e14718900f2b56c6abea59 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/38eb9f2769e14718900f2b56c6abea59 2023-04-27 03:55:40,917 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/38eb9f2769e14718900f2b56c6abea59, entries=150, sequenceid=177, filesize=11.8 K 2023-04-27 03:55:40,918 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/19974a6638f34a6187fad2e600337ff5 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/19974a6638f34a6187fad2e600337ff5 2023-04-27 03:55:40,924 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/19974a6638f34a6187fad2e600337ff5, entries=150, sequenceid=177, filesize=11.8 K 2023-04-27 03:55:40,937 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/8a6b439808544a3ebe2377ccdbd47ed2 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/8a6b439808544a3ebe2377ccdbd47ed2 2023-04-27 03:55:40,943 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/8a6b439808544a3ebe2377ccdbd47ed2, entries=150, sequenceid=177, filesize=11.8 K 2023-04-27 03:55:40,965 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 98cbc3f1a78abce9e99d0349272cd6a4 in 634ms, sequenceid=177, compaction requested=false 2023-04-27 03:55:40,965 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:40,965 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:40,965 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:40,966 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:40,966 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,966 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:40,966 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:40,967 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:40,967 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:40,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:40,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:40,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:40,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:40,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:40,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:40,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:40,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:40,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30fb0775[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:40,969 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:40,969 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:40,970 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:40,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:40,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:40,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:40,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:40,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:40,971 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,972 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:40,976 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:40,976 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:40,976 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:40,976 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:40,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:40,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:40,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:40,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:40,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:40,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:40,979 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:40,979 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:40,979 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:40,979 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:40,979 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:40,981 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,981 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:40,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:40,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,982 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:40,982 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,987 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:40,988 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:40,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:40,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:40,988 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,988 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:40,988 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:40,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:40,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:40,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:40,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:40,988 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:40,988 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:40,988 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:40,988 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:40,988 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:40,988 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:40,988 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,042 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-27 03:55:41,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:41,043 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:41,043 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:41,043 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:41,043 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:41,043 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:41,043 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:41,069 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/2f66038a23574b7f86fba7d5e7409021 2023-04-27 03:55:41,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:41,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567801072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:41,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567801073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:41,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567801076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:41,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567801076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:41,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567801076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,088 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:41,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:41,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:41,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:41,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:41,105 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:41,105 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:41,105 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:41,105 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:41,105 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,105 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,107 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,107 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:41,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:41,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:41,107 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,107 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:41,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:41,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:41,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:41,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:41,111 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:41,111 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:41,111 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:41,111 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:41,111 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:41,111 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/a453869a5d9a4172adff2fd35412eb58 2023-04-27 03:55:41,111 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:41,112 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:41,112 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:41,112 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:41,112 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:41,112 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:41,112 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,112 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:41,112 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:41,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,114 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:41,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:41,115 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,115 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:41,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:41,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:41,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:41,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:41,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26dafbaf[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:41,120 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:41,120 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,122 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,122 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:41,122 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,124 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:41,124 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:41,124 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:41,125 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:41,125 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:41,125 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,125 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:41,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:41,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:41,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:41,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:41,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:41,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:41,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:41,128 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:41,128 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:41,128 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:41,128 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:41,128 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:41,130 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,130 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:41,130 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:41,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:41,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,130 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:41,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:41,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:41,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:41,131 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:41,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,135 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:41,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:41,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:41,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:41,136 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:41,136 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:41,139 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,140 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,140 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,140 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,140 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,140 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,140 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,140 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:41,140 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:41,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:41,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:41,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:41,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:41,178 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/365cb20a073049b2a87c1c3af1111c25 2023-04-27 03:55:41,182 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/2f66038a23574b7f86fba7d5e7409021 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/2f66038a23574b7f86fba7d5e7409021 2023-04-27 03:55:41,186 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/2f66038a23574b7f86fba7d5e7409021, entries=250, sequenceid=194, filesize=16.4 K 2023-04-27 03:55:41,186 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/a453869a5d9a4172adff2fd35412eb58 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/a453869a5d9a4172adff2fd35412eb58 2023-04-27 03:55:41,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:41,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567801178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,191 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/a453869a5d9a4172adff2fd35412eb58, entries=150, sequenceid=194, filesize=11.8 K 2023-04-27 03:55:41,191 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/365cb20a073049b2a87c1c3af1111c25 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/365cb20a073049b2a87c1c3af1111c25 2023-04-27 03:55:41,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:41,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567801186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,195 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/365cb20a073049b2a87c1c3af1111c25, entries=150, sequenceid=194, filesize=11.8 K 2023-04-27 03:55:41,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:41,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567801188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,196 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 98cbc3f1a78abce9e99d0349272cd6a4 in 154ms, sequenceid=194, compaction requested=true 2023-04-27 03:55:41,196 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:41,196 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:41,196 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:41,196 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:41,196 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-27 03:55:41,196 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:41,198 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-27 03:55:41,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:41,198 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40991 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:41,198 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/A is initiating minor compaction (all files) 2023-04-27 03:55:41,198 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/A in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:41,198 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/97ef251156094006bf37bf156e1825b4, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/38eb9f2769e14718900f2b56c6abea59, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/2f66038a23574b7f86fba7d5e7409021] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=40.0 K 2023-04-27 03:55:41,199 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36211 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:41,199 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/B is initiating minor compaction (all files) 2023-04-27 03:55:41,199 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/B in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:41,199 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/d6b6b1f57c67492d860036d1f5325f94, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/19974a6638f34a6187fad2e600337ff5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/a453869a5d9a4172adff2fd35412eb58] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=35.4 K 2023-04-27 03:55:41,200 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 97ef251156094006bf37bf156e1825b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682567739108 2023-04-27 03:55:41,200 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting d6b6b1f57c67492d860036d1f5325f94, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682567739108 2023-04-27 03:55:41,198 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:41,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:41,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:41,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:41,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:41,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:41,201 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 19974a6638f34a6187fad2e600337ff5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1682567739754 2023-04-27 03:55:41,201 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 38eb9f2769e14718900f2b56c6abea59, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1682567739754 2023-04-27 03:55:41,202 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting a453869a5d9a4172adff2fd35412eb58, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1682567740401 2023-04-27 03:55:41,202 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 2f66038a23574b7f86fba7d5e7409021, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1682567740397 2023-04-27 03:55:41,228 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#A#compaction#435 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:41,236 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:41,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:41,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:41,241 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#B#compaction#436 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:41,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:41,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:41,248 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:41,248 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:41,248 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:41,248 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:41,249 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,249 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,250 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,250 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:41,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:41,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:41,250 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,250 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:41,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:41,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:41,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:41,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567801243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:41,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567801245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,266 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:41,266 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:41,266 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:41,266 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:41,266 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:41,271 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:41,271 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:41,271 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:41,271 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:41,271 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:41,272 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:41,272 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:41,272 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:41,272 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,272 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:41,272 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:41,274 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,274 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:41,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:41,275 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,275 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:41,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:41,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:41,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:41,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:41,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77faa46[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:41,276 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:41,276 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,277 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:41,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:41,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:41,278 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,278 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:41,281 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:41,281 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:41,281 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:41,282 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:41,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:41,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:41,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:41,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:41,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:41,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:41,284 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:41,284 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:41,284 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:41,284 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:41,284 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:41,286 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,287 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,287 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:41,287 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:41,287 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:41,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:41,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:41,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:41,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:41,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,290 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:41,290 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:41,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:41,290 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,290 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,290 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,290 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:41,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:41,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:41,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:41,290 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,291 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,291 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,290 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:41,291 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:41,291 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:41,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:41,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:41,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:41,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:41,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567801354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,366 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:41,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567801355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,391 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:41,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:41,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:41,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:41,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:41,394 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:41,394 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:41,394 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:41,394 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:41,395 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,395 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,397 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:41,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:41,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:41,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,399 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:41,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,399 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,400 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:41,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:41,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:41,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567801394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,405 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:41,405 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:41,405 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:41,405 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:41,405 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:41,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:41,408 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:41,408 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:41,408 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:41,408 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:41,408 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:41,408 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:41,408 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:41,408 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:41,408 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,408 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:41,408 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:41,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567801399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:41,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567801399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,411 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:41,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:41,411 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:41,411 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,411 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:41,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:41,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:41,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:41,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@354494f6[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:41,413 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:41,413 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,415 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,415 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:41,415 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:41,415 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:41,416 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,416 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:41,417 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:41,417 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:41,417 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:41,417 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:41,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:41,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:41,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:41,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:41,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:41,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:41,419 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:41,419 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:41,419 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:41,419 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:41,419 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:41,422 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,422 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:41,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,422 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:41,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:41,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:41,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:41,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:41,425 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:41,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,427 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:41,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,433 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:41,433 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:41,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:41,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:41,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:41,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:41,433 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,433 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,434 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,434 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:41,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:41,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:41,434 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,434 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,434 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,434 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,434 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:41,434 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:41,534 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:41,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:41,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:41,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:41,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:41,539 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:41,539 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:41,539 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:41,539 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:41,539 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,539 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,541 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:41,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:41,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:41,541 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,543 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,544 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:41,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:41,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,544 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:41,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:41,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:41,548 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:41,548 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:41,548 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:41,551 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:41,551 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:41,551 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:41,551 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:41,551 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:41,551 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:41,551 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:41,551 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:41,551 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,551 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:41,552 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:41,553 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:41,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:41,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:41,554 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:41,555 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,555 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:41,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:41,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:41,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5deced84[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:41,556 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:41,556 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,559 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,559 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:41,559 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:41,559 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:41,559 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,559 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:41,566 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:41,567 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:41,567 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:41,567 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:41,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:41,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:41,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:41,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:41,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:41,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567801568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:41,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567801569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:41,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:41,581 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:41,581 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:41,581 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:41,581 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:41,581 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:41,583 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,583 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:41,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:41,583 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:41,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:41,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:41,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:41,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,584 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:41,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:41,587 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,589 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:41,589 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:41,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:41,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:41,590 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,590 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,590 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,590 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,590 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,590 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,590 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,590 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:41,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:41,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:41,590 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:41,590 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:41,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:41,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:41,672 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/d3c3e9a063c94dc5a77e91c5f19efb11 2023-04-27 03:55:41,677 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/5e604011c20b4cdf970951646855ceb3 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/5e604011c20b4cdf970951646855ceb3 2023-04-27 03:55:41,690 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:41,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:41,692 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/a14b8d58f9004774ac184eff7d961410 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/a14b8d58f9004774ac184eff7d961410 2023-04-27 03:55:41,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:41,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:41,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:41,712 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:41,713 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:41,713 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:41,713 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:41,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:41,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567801702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,714 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,714 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,716 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:41,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:41,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:41,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,717 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:41,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,718 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,718 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:41,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:41,719 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/B of 98cbc3f1a78abce9e99d0349272cd6a4 into 5e604011c20b4cdf970951646855ceb3(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:41,719 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:41,719 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/B, priority=13, startTime=1682567741196; duration=0sec 2023-04-27 03:55:41,719 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:41,720 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:41,721 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/A of 98cbc3f1a78abce9e99d0349272cd6a4 into a14b8d58f9004774ac184eff7d961410(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:41,721 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:41,721 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/A, priority=13, startTime=1682567741196; duration=0sec 2023-04-27 03:55:41,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:41,721 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:41,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:41,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:41,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:41,721 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:41,722 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36211 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:41,722 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/C is initiating minor compaction (all files) 2023-04-27 03:55:41,722 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/C in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:41,722 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/6eb198f1655e429a813e22a3852428ae, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/8a6b439808544a3ebe2377ccdbd47ed2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/365cb20a073049b2a87c1c3af1111c25] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=35.4 K 2023-04-27 03:55:41,722 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 6eb198f1655e429a813e22a3852428ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682567739108 2023-04-27 03:55:41,723 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 8a6b439808544a3ebe2377ccdbd47ed2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1682567739754 2023-04-27 03:55:41,723 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 365cb20a073049b2a87c1c3af1111c25, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1682567740401 2023-04-27 03:55:41,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:41,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:41,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567801710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567801713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,736 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:41,736 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:41,736 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/3f2c5ff6759c4c8094668bb6b00c8136 2023-04-27 03:55:41,736 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:41,736 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:41,737 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:41,737 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:41,737 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:41,737 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:41,737 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,745 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#C#compaction#438 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:41,757 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:41,757 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:41,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,760 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:41,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:41,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:41,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:41,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:41,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:41,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:41,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c1a7215[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:41,762 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:41,762 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,763 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,764 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:41,764 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:41,764 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,764 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:41,764 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:41,765 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:41,765 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:41,766 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:41,766 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:41,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:41,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:41,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:41,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:41,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:41,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:41,769 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:41,769 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:41,769 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:41,769 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:41,769 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:41,771 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,771 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:41,771 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,772 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:41,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:41,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:41,772 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:41,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:41,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:41,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:41,776 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:41,776 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:41,776 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:41,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:41,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:41,776 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:41,776 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,776 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,776 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:41,776 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:41,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:41,776 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:41,777 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,777 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,777 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:41,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:41,877 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:41,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:41,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:41,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:41,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567801877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:41,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567801878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:41,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:41,889 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:41,889 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:41,889 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:41,889 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:41,889 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,889 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,891 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,891 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:41,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:41,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:41,891 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,891 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:41,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:41,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,892 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:41,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:41,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:41,896 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:41,896 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:41,896 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:41,904 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:41,904 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:41,904 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:41,904 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:41,904 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:41,904 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:41,904 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:41,904 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:41,904 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,904 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:41,904 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:41,906 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,906 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:41,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:41,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:41,907 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,907 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:41,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:41,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:41,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:41,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@564dc04a[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:41,908 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:41,908 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,912 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,912 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:41,912 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:41,912 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:41,912 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,912 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:41,914 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:41,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:41,914 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:41,915 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:41,915 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:41,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:41,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:41,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:41,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:41,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:41,917 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:41,917 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:41,917 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:41,917 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:41,917 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:41,919 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,919 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:41,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,920 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:41,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:41,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:41,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:41,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:41,921 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:41,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:41,923 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,927 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:41,927 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:41,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:41,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:41,927 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,927 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,927 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:41,927 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:41,927 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:41,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:41,928 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:41,928 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:41,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:41,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:41,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:41,927 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:41,928 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:41,928 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:41,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:42,028 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:42,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:42,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:42,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:42,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:42,042 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:42,042 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:42,042 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:42,042 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:42,044 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,044 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,046 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:42,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:42,046 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:42,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,047 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,047 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:42,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:42,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,048 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:42,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:42,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:42,052 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:42,052 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:42,052 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:42,057 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:42,057 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:42,057 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:42,057 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:42,057 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:42,057 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:42,058 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:42,058 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:42,058 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,058 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:42,058 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:42,059 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,059 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:42,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:42,060 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:42,060 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:42,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:42,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:42,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:42,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@473f0d51[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:42,061 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:42,061 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,062 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,063 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:42,063 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:42,063 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:42,067 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,067 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:42,068 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:42,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:42,068 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:42,068 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:42,068 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:42,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:42,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:42,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:42,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:42,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:42,072 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:42,073 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:42,073 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:42,073 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:42,073 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:42,075 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,075 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:42,075 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:42,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:42,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:42,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:42,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:42,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:42,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,079 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:42,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,082 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,085 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:42,085 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:42,085 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,085 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:42,085 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:42,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:42,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:42,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:42,085 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,085 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,085 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,085 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,085 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:42,085 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:42,085 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:42,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:42,185 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:42,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:42,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:42,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:42,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:42,191 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:42,191 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:42,192 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:42,192 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:42,192 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,192 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,193 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:42,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:42,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:42,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,195 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:42,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:42,195 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,195 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:42,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:42,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:42,196 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:42,196 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:42,196 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:42,200 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:42,200 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:42,204 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:42,204 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:42,204 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:42,204 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:42,204 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:42,204 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:42,204 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,204 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:42,204 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:42,207 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,207 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:42,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:42,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,210 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,210 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:42,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:42,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:42,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d0558ff[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:42,211 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:42,211 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,211 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/a1019f6cf2a44d97b598f0aeee6fcced 2023-04-27 03:55:42,215 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,216 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:42,216 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:42,216 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:42,216 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/d3c3e9a063c94dc5a77e91c5f19efb11 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/d3c3e9a063c94dc5a77e91c5f19efb11 2023-04-27 03:55:42,218 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:42,218 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:42,218 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:42,219 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,219 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:42,219 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:42,219 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:42,219 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:42,219 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:42,219 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:42,219 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:42,221 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,221 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:42,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:42,221 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:42,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,221 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/d3c3e9a063c94dc5a77e91c5f19efb11, entries=200, sequenceid=213, filesize=14.1 K 2023-04-27 03:55:42,222 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:42,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:42,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:42,223 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/3f2c5ff6759c4c8094668bb6b00c8136 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/3f2c5ff6759c4c8094668bb6b00c8136 2023-04-27 03:55:42,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:42,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:42,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,224 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567802219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:42,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,228 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/3f2c5ff6759c4c8094668bb6b00c8136, entries=150, sequenceid=213, filesize=11.8 K 2023-04-27 03:55:42,228 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:42,228 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:42,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:42,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:42,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:42,228 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,229 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,229 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,228 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:42,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:42,229 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:42,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:42,229 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:42,229 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:42,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:42,229 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,229 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,229 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,230 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/a1019f6cf2a44d97b598f0aeee6fcced as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/a1019f6cf2a44d97b598f0aeee6fcced 2023-04-27 03:55:42,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:42,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567802229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,242 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/a1019f6cf2a44d97b598f0aeee6fcced, entries=150, sequenceid=213, filesize=11.8 K 2023-04-27 03:55:42,244 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 98cbc3f1a78abce9e99d0349272cd6a4 in 1046ms, sequenceid=213, compaction requested=false 2023-04-27 03:55:42,244 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:42,244 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/f09518bafff04883bc48eb6460045db3 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/f09518bafff04883bc48eb6460045db3 2023-04-27 03:55:42,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:42,245 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-27 03:55:42,246 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:42,246 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:42,246 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:42,246 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:42,246 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:42,246 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:42,253 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/C of 98cbc3f1a78abce9e99d0349272cd6a4 into f09518bafff04883bc48eb6460045db3(size=12.0 K), total size for store is 23.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:42,253 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:42,253 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/C, priority=13, startTime=1682567741196; duration=0sec 2023-04-27 03:55:42,253 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:42,274 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/b8daaed48fb94e6c8f60731cc39d6083 2023-04-27 03:55:42,291 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/1292fcab792f4876a3809a42313159d5 2023-04-27 03:55:42,329 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:42,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:42,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:42,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:42,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:42,332 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:42,332 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:42,332 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:42,332 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:42,333 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,333 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,335 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:42,335 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:42,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:42,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,336 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,336 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:42,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:42,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,336 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:42,336 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:42,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:42,337 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:42,337 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:42,337 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:42,337 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:42,337 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:42,337 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:42,337 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:42,337 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:42,337 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:42,337 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:42,337 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:42,337 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,337 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:42,337 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:42,339 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,339 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:42,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:42,340 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:42,340 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:42,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:42,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:42,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:42,341 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:42,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2700790[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:42,341 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,342 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,342 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:42,342 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:42,342 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:42,342 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,342 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:42,343 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:42,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:42,344 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:42,344 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:42,344 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:42,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:42,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:42,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:42,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:42,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:42,346 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:42,346 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:42,346 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:42,346 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:42,346 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:42,349 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,349 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:42,349 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:42,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:42,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:42,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:42,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:42,349 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:42,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,350 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:42,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,352 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,352 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,352 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,352 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,352 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,352 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,352 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:42,352 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:42,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:42,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:42,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:42,352 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:42,353 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:42,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:42,353 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:42,353 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:42,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:42,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:42,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:42,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567802379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:42,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567802385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:42,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567802388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,453 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:42,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:42,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:42,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:42,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:42,456 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:42,456 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:42,456 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:42,456 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:42,457 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,457 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,461 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:42,461 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:42,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:42,461 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,461 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:42,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:42,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,462 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:42,471 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:42,471 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:42,471 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:42,471 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:42,471 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:42,476 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:42,476 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:42,476 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:42,476 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:42,476 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:42,476 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:42,476 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:42,476 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:42,476 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,476 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:42,477 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:42,478 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:42,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:42,478 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:42,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:42,479 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,479 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:42,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:42,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:42,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a07c804[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:42,482 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:42,482 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:42,485 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,485 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:42,485 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:42,485 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:42,487 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:42,487 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:42,487 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:42,488 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:42,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:42,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:42,491 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:42,491 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:42,491 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:42,491 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:42,491 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:42,492 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,492 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:42,492 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:42,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:42,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:42,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:42,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:42,494 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:42,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:42,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567802488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,495 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:42,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,499 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:42,499 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:42,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:42,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:42,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:42,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:42,499 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,499 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,500 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,500 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:42,500 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:42,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:42,500 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,500 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,500 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,500 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:42,500 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:42,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:42,600 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:42,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:42,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:42,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:42,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:42,614 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:42,614 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:42,614 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:42,614 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:42,614 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,614 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:42,616 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:42,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:42,617 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,617 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:42,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:42,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:42,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:42,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:42,619 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:42,619 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:42,619 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:42,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:42,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:42,620 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:42,620 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:42,620 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:42,620 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:42,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:42,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:42,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:42,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:42,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:42,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:42,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:42,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:42,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:42,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:42,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56d0f90[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:42,624 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:42,624 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,626 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,626 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:42,628 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,629 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,629 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,629 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:42,629 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:42,629 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:42,629 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:42,630 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:42,630 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,630 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:42,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:42,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:42,630 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:42,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:42,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:42,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:42,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:42,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:42,633 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:42,633 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:42,633 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:42,633 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:42,633 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:42,637 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,637 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:42,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,638 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:42,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:42,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:42,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:42,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:42,640 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:42,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,641 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:42,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:42,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,646 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,646 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,646 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,646 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,646 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:42,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:42,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:42,646 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:42,646 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,646 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,646 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:42,646 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:42,646 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:42,646 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:42,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:42,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:42,646 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:42,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:42,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:42,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:42,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567802697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,716 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/a49da27384144a3297a36166a2c6f597 2023-04-27 03:55:42,720 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/b8daaed48fb94e6c8f60731cc39d6083 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/b8daaed48fb94e6c8f60731cc39d6083 2023-04-27 03:55:42,723 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/b8daaed48fb94e6c8f60731cc39d6083, entries=200, sequenceid=229, filesize=14.1 K 2023-04-27 03:55:42,724 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/1292fcab792f4876a3809a42313159d5 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/1292fcab792f4876a3809a42313159d5 2023-04-27 03:55:42,729 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/1292fcab792f4876a3809a42313159d5, entries=150, sequenceid=229, filesize=11.8 K 2023-04-27 03:55:42,730 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/a49da27384144a3297a36166a2c6f597 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/a49da27384144a3297a36166a2c6f597 2023-04-27 03:55:42,733 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/a49da27384144a3297a36166a2c6f597, entries=150, sequenceid=229, filesize=11.8 K 2023-04-27 03:55:42,734 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 98cbc3f1a78abce9e99d0349272cd6a4 in 489ms, sequenceid=229, compaction requested=true 2023-04-27 03:55:42,734 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:42,734 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:42,734 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:42,734 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:42,734 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:42,735 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:42,736 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41094 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:42,737 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/A is initiating minor compaction (all files) 2023-04-27 03:55:42,737 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:42,737 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/A in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:42,737 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/B is initiating minor compaction (all files) 2023-04-27 03:55:42,737 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/a14b8d58f9004774ac184eff7d961410, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/d3c3e9a063c94dc5a77e91c5f19efb11, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/b8daaed48fb94e6c8f60731cc39d6083] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=40.1 K 2023-04-27 03:55:42,737 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/B in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:42,737 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/5e604011c20b4cdf970951646855ceb3, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/3f2c5ff6759c4c8094668bb6b00c8136, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/1292fcab792f4876a3809a42313159d5] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=35.5 K 2023-04-27 03:55:42,737 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting a14b8d58f9004774ac184eff7d961410, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1682567740401 2023-04-27 03:55:42,738 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 5e604011c20b4cdf970951646855ceb3, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1682567740401 2023-04-27 03:55:42,738 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting d3c3e9a063c94dc5a77e91c5f19efb11, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682567741055 2023-04-27 03:55:42,738 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 3f2c5ff6759c4c8094668bb6b00c8136, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682567741055 2023-04-27 03:55:42,738 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting b8daaed48fb94e6c8f60731cc39d6083, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1682567741226 2023-04-27 03:55:42,738 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 1292fcab792f4876a3809a42313159d5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1682567741226 2023-04-27 03:55:42,746 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:42,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:42,748 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#B#compaction#443 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:42,748 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#A#compaction#444 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:42,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:42,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:42,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:42,761 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:42,761 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:42,761 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:42,761 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:42,762 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,762 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,763 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,763 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:42,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:42,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:42,763 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:42,763 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:42,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:42,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:42,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:42,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:42,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:42,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:42,773 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:42,773 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:42,773 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:42,780 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:42,780 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:42,780 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:42,780 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:42,780 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-27 03:55:42,780 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:42,781 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:42,781 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:42,781 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:42,781 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:42,781 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:42,782 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/8f83895db57648f0a931ad5aa505de26 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/8f83895db57648f0a931ad5aa505de26 2023-04-27 03:55:42,788 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/B of 98cbc3f1a78abce9e99d0349272cd6a4 into 8f83895db57648f0a931ad5aa505de26(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:42,788 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:42,788 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/B, priority=13, startTime=1682567742734; duration=0sec 2023-04-27 03:55:42,788 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:42,788 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:42,789 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:42,789 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/C is initiating minor compaction (all files) 2023-04-27 03:55:42,789 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/C in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:42,789 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/f09518bafff04883bc48eb6460045db3, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/a1019f6cf2a44d97b598f0aeee6fcced, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/a49da27384144a3297a36166a2c6f597] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=35.5 K 2023-04-27 03:55:42,789 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting f09518bafff04883bc48eb6460045db3, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1682567740401 2023-04-27 03:55:42,790 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting a1019f6cf2a44d97b598f0aeee6fcced, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682567741055 2023-04-27 03:55:42,790 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting a49da27384144a3297a36166a2c6f597, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1682567741226 2023-04-27 03:55:42,791 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/a4bf7b54ee15466aa49c91d0e9f0172f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/a4bf7b54ee15466aa49c91d0e9f0172f 2023-04-27 03:55:42,796 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/A of 98cbc3f1a78abce9e99d0349272cd6a4 into a4bf7b54ee15466aa49c91d0e9f0172f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:42,796 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:42,796 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/A, priority=13, startTime=1682567742734; duration=0sec 2023-04-27 03:55:42,796 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:42,797 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#C#compaction#446 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:43,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:43,006 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:43,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:43,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567803065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:43,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567803171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,203 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/345246d6362b45dea1eb62a9b6ee48c3 2023-04-27 03:55:43,207 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/491847d70a67486e8339d9946ee3bbbd as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/491847d70a67486e8339d9946ee3bbbd 2023-04-27 03:55:43,217 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/C of 98cbc3f1a78abce9e99d0349272cd6a4 into 491847d70a67486e8339d9946ee3bbbd(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:43,217 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:43,217 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/C, priority=13, startTime=1682567742735; duration=0sec 2023-04-27 03:55:43,217 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:43,219 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/dddb0b9c6d0645189cd85622b8c2b186 2023-04-27 03:55:43,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:43,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567803235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,249 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/8f076a24ebe3479e9ba66c6adafc777c 2023-04-27 03:55:43,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:43,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567803246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,256 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/345246d6362b45dea1eb62a9b6ee48c3 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/345246d6362b45dea1eb62a9b6ee48c3 2023-04-27 03:55:43,260 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/345246d6362b45dea1eb62a9b6ee48c3, entries=150, sequenceid=248, filesize=11.8 K 2023-04-27 03:55:43,262 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/dddb0b9c6d0645189cd85622b8c2b186 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/dddb0b9c6d0645189cd85622b8c2b186 2023-04-27 03:55:43,266 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/dddb0b9c6d0645189cd85622b8c2b186, entries=150, sequenceid=248, filesize=11.8 K 2023-04-27 03:55:43,267 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/8f076a24ebe3479e9ba66c6adafc777c as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/8f076a24ebe3479e9ba66c6adafc777c 2023-04-27 03:55:43,276 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/8f076a24ebe3479e9ba66c6adafc777c, entries=150, sequenceid=248, filesize=11.8 K 2023-04-27 03:55:43,277 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 98cbc3f1a78abce9e99d0349272cd6a4 in 497ms, sequenceid=248, compaction requested=false 2023-04-27 03:55:43,277 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:43,277 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:43,277 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:43,277 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:43,277 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,280 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:43,280 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:43,281 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:43,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:43,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:43,282 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,282 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,283 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:43,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:43,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:43,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:43,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52f87d62[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:43,284 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:43,284 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,286 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,287 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:43,287 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:43,287 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:43,288 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:43,288 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:43,288 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:43,289 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,289 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:43,289 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:43,289 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:43,289 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:43,289 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:43,289 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:43,289 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:43,293 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,293 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,293 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:43,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:43,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:43,293 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:43,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:43,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,294 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:43,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:43,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:43,297 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:43,297 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,297 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,297 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,298 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,298 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,298 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,298 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,298 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:43,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:43,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:43,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:43,298 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:43,298 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:43,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:43,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:43,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:43,381 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-27 03:55:43,381 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:43,381 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:43,381 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:43,381 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:43,381 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:43,381 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:43,395 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/8b03cffb0ac9490b80f417d63a3b98d6 2023-04-27 03:55:43,398 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:43,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:43,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:43,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:43,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:43,403 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:43,404 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:43,404 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:43,404 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:43,406 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,406 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,408 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:43,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:43,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:43,408 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:43,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:43,410 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,410 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:43,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:43,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:43,412 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:43,412 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:43,412 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:43,416 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:43,416 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:43,416 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:43,416 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:43,416 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:43,416 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:43,416 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:43,416 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:43,416 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,416 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:43,416 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:43,419 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/cd0962cedc8c4f6a9bcbdc07f8233073 2023-04-27 03:55:43,420 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,420 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:43,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:43,421 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,421 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:43,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:43,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:43,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:43,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50b6ac2a[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:43,422 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:43,422 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,424 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,424 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:43,424 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:43,424 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:43,424 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,424 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:43,427 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:43,427 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:43,427 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:43,427 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:43,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:43,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:43,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:43,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:43,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:43,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567803426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:43,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:43,430 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:43,430 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:43,430 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:43,430 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:43,430 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:43,432 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,432 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,432 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:43,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:43,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:43,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,432 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:43,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:43,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:43,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,434 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:43,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:43,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:43,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45932 deadline: 1682567803427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:43,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567803428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:43,436 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:43,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:43,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:43,436 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:43,436 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,437 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,437 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:43,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:43,437 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:43,437 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,437 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:43,437 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:43,437 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:43,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:43,437 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,438 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,442 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/ab20fab8f0994609b05b06f45d965029 2023-04-27 03:55:43,447 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/8b03cffb0ac9490b80f417d63a3b98d6 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/8b03cffb0ac9490b80f417d63a3b98d6 2023-04-27 03:55:43,452 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/8b03cffb0ac9490b80f417d63a3b98d6, entries=150, sequenceid=266, filesize=11.9 K 2023-04-27 03:55:43,453 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/cd0962cedc8c4f6a9bcbdc07f8233073 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/cd0962cedc8c4f6a9bcbdc07f8233073 2023-04-27 03:55:43,457 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/cd0962cedc8c4f6a9bcbdc07f8233073, entries=150, sequenceid=266, filesize=11.9 K 2023-04-27 03:55:43,458 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/ab20fab8f0994609b05b06f45d965029 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/ab20fab8f0994609b05b06f45d965029 2023-04-27 03:55:43,461 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/ab20fab8f0994609b05b06f45d965029, entries=150, sequenceid=266, filesize=11.9 K 2023-04-27 03:55:43,462 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 98cbc3f1a78abce9e99d0349272cd6a4 in 81ms, sequenceid=266, compaction requested=true 2023-04-27 03:55:43,462 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:43,462 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:43,462 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:43,462 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:43,462 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:43,462 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:43,465 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36566 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:43,465 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/B is initiating minor compaction (all files) 2023-04-27 03:55:43,465 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/B in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:43,465 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/8f83895db57648f0a931ad5aa505de26, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/dddb0b9c6d0645189cd85622b8c2b186, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/cd0962cedc8c4f6a9bcbdc07f8233073] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=35.7 K 2023-04-27 03:55:43,465 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36566 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:43,465 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/A is initiating minor compaction (all files) 2023-04-27 03:55:43,465 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/A in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:43,466 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/a4bf7b54ee15466aa49c91d0e9f0172f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/345246d6362b45dea1eb62a9b6ee48c3, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/8b03cffb0ac9490b80f417d63a3b98d6] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=35.7 K 2023-04-27 03:55:43,466 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 8f83895db57648f0a931ad5aa505de26, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1682567741226 2023-04-27 03:55:43,466 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting a4bf7b54ee15466aa49c91d0e9f0172f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1682567741226 2023-04-27 03:55:43,467 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting dddb0b9c6d0645189cd85622b8c2b186, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1682567742281 2023-04-27 03:55:43,467 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 345246d6362b45dea1eb62a9b6ee48c3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1682567742281 2023-04-27 03:55:43,467 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting cd0962cedc8c4f6a9bcbdc07f8233073, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1682567743034 2023-04-27 03:55:43,467 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 8b03cffb0ac9490b80f417d63a3b98d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1682567743034 2023-04-27 03:55:43,475 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#B#compaction#452 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:43,479 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#A#compaction#453 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:43,529 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/74a1eac81ab645b1b814c567c790833c as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/74a1eac81ab645b1b814c567c790833c 2023-04-27 03:55:43,533 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/A of 98cbc3f1a78abce9e99d0349272cd6a4 into 74a1eac81ab645b1b814c567c790833c(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:43,534 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:43,534 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/A, priority=13, startTime=1682567743462; duration=0sec 2023-04-27 03:55:43,534 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:43,534 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:43,535 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36566 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:43,535 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/C is initiating minor compaction (all files) 2023-04-27 03:55:43,535 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/C in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:43,535 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/491847d70a67486e8339d9946ee3bbbd, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/8f076a24ebe3479e9ba66c6adafc777c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/ab20fab8f0994609b05b06f45d965029] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=35.7 K 2023-04-27 03:55:43,535 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 491847d70a67486e8339d9946ee3bbbd, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1682567741226 2023-04-27 03:55:43,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:43,536 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-27 03:55:43,536 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:43,536 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:43,536 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:43,536 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:43,536 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:43,536 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:43,536 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 8f076a24ebe3479e9ba66c6adafc777c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1682567742281 2023-04-27 03:55:43,537 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:43,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:43,539 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting ab20fab8f0994609b05b06f45d965029, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1682567743034 2023-04-27 03:55:43,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:43,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:43,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:43,543 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:43,543 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:43,543 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:43,543 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:43,543 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,543 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,544 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,545 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,545 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:43,546 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#C#compaction#455 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:43,547 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:43,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:43,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:43,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:43,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:43,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:43,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:43,564 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:43,564 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:43,564 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:43,568 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:43,568 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:43,568 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:43,568 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:43,568 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:43,568 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:43,568 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:43,568 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:43,568 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,574 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:43,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567803566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:43,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567803571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:43,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567803573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,585 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:43,585 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:43,588 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,588 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:43,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:43,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:43,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:43,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:43,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:43,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@547519d0[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:43,588 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,596 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:43,590 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:43,596 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,599 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,599 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:43,600 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,600 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:43,601 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:43,601 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:43,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:43,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:43,602 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,602 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:43,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:43,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:43,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:43,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:43,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:43,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:43,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:43,604 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:43,604 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:43,604 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:43,604 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:43,604 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:43,606 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:43,606 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,606 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,606 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:43,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:43,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:43,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:43,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,607 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:43,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:43,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:43,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:43,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:43,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:43,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:43,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,612 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:43,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,612 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:43,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:43,612 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:43,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:43,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:43,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567803676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,685 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:43,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567803681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:43,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567803682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,712 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:43,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:43,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:43,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:43,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:43,716 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:43,716 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:43,716 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:43,716 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:43,716 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,716 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,717 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:43,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:43,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:43,718 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,718 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,718 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:43,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:43,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:43,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:43,719 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:43,719 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:43,719 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:43,719 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:43,719 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:43,719 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:43,719 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:43,719 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:43,720 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:43,720 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:43,720 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:43,720 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:43,720 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,720 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:43,720 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:43,722 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:43,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,722 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:43,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:43,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:43,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:43,724 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:43,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39898d41[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:43,724 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,725 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,725 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:43,726 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,726 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:43,726 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:43,726 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:43,728 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:43,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:43,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:43,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:43,731 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:43,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:43,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:43,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:43,731 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:43,733 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:43,733 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,733 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:43,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:43,734 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,735 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,739 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:43,739 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:43,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:43,740 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,740 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,740 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,740 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,740 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:43,740 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,740 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,740 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:43,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:43,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:43,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:43,741 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:43,741 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:43,841 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:43,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:43,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:43,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:43,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:43,856 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:43,856 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:43,856 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:43,856 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:43,857 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,857 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,858 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,858 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:43,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:43,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:43,858 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,858 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:43,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:43,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:43,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:43,859 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:43,860 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:43,860 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:43,860 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:43,861 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:43,861 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:43,861 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:43,861 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:43,861 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:43,862 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:43,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:43,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:43,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:43,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:43,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,864 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:43,865 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,865 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:43,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:43,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:43,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:43,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:43,866 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:43,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@288be47f[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:43,866 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,867 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,868 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:43,868 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:43,868 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:43,869 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,869 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:43,874 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:43,874 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:43,874 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:43,874 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:43,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:43,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:43,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:43,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:43,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:43,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:43,878 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:43,878 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:43,878 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:43,878 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:43,878 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:43,881 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,881 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:43,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,881 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:43,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:43,882 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,884 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:43,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:43,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:43,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567803881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,886 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:43,886 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:43,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:43,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:43,886 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,886 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,886 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,886 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,886 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,886 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:43,886 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,886 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:43,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:43,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:43,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:43,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:43,887 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:43,887 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:43,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:43,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567803887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:43,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567803890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,904 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/97c4ba2b20064700801b023518d64de7 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/97c4ba2b20064700801b023518d64de7 2023-04-27 03:55:43,909 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/B of 98cbc3f1a78abce9e99d0349272cd6a4 into 97c4ba2b20064700801b023518d64de7(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:43,909 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:43,909 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/B, priority=13, startTime=1682567743462; duration=0sec 2023-04-27 03:55:43,909 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:43,981 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/882d1ebb8fa049c58e1aa3b44f352f46 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/882d1ebb8fa049c58e1aa3b44f352f46 2023-04-27 03:55:43,987 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:43,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:43,988 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/5074919511504133a6f3abcf7a731f4f 2023-04-27 03:55:43,989 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/C of 98cbc3f1a78abce9e99d0349272cd6a4 into 882d1ebb8fa049c58e1aa3b44f352f46(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:43,989 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:43,989 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/C, priority=13, startTime=1682567743462; duration=0sec 2023-04-27 03:55:43,989 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:43,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:43,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:43,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:43,992 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:43,992 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:43,992 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:43,992 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:43,993 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,993 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,994 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:43,994 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:43,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:43,994 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:43,994 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:43,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:43,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:44,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:44,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:44,000 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:44,000 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:44,000 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:44,003 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:44,003 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:44,003 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:44,003 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:44,003 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:44,003 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:44,003 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:44,003 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:44,004 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,008 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:44,008 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:44,009 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,009 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:44,010 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,010 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:44,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:44,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:44,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d39dab8[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:44,013 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:44,013 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,015 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,016 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:44,016 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:44,016 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:44,021 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:44,021 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:44,021 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:44,022 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,022 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:44,022 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:44,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:44,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:44,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:44,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:44,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:44,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:44,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:44,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:44,026 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:44,026 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:44,026 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:44,026 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:44,026 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:44,027 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,028 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:44,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:44,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:44,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,030 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,030 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:44,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:44,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:44,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:44,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:44,033 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:44,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:44,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:44,038 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:44,038 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:44,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:44,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:44,038 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,038 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:44,038 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:44,038 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,038 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:44,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:44,038 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,038 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:44,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:44,038 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:44,039 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:44,039 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:44,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:44,139 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:44,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:44,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:44,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:44,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:44,143 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:44,143 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:44,143 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:44,143 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:44,143 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,143 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:44,146 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,146 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:44,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:44,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:44,147 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,147 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:44,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:44,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:44,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:44,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:44,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:44,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:44,148 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:44,148 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:44,148 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:44,148 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:44,148 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:44,148 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:44,149 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:44,149 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:44,149 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:44,149 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:44,149 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:44,149 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,149 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:44,149 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:44,150 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,150 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:44,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:44,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:44,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:44,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:44,154 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:44,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@317ee156[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:44,154 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,155 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,155 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:44,155 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:44,155 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:44,156 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,156 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:44,158 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:44,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:44,158 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:44,158 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:44,158 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:44,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:44,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:44,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:44,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:44,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:44,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:44,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:44,160 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:44,160 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:44,160 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:44,160 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:44,160 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:44,167 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,167 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:44,167 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:44,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:44,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:44,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:44,168 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:44,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:44,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:44,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:44,169 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:44,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:44,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,172 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,172 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:44,172 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:44,172 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:44,172 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:44,172 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:44,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:44,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:44,172 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:44,172 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,173 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,173 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,173 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:44,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:44,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:44,173 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:44,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:44,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:44,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:44,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567804188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:44,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567804198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:44,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567804200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,273 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:44,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:44,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:44,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:44,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:44,277 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:44,277 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:44,277 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:44,277 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:44,278 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,278 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:44,279 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:44,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:44,279 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:44,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,280 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,280 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:44,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:44,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:44,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:44,285 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:44,285 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:44,285 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:44,285 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:44,285 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:44,285 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:44,285 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:44,285 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:44,286 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:44,286 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:44,286 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:44,286 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,286 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:44,286 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:44,287 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,287 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:44,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:44,288 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,288 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:44,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:44,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:44,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:44,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:44,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:44,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@446881bd[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:44,289 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:44,289 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,292 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,292 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:44,292 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:44,292 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:44,292 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,292 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:44,293 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:44,294 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:44,294 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:44,294 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:44,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:44,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:44,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:44,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:44,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:44,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:44,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:44,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:44,298 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:44,298 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:44,298 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:44,298 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:44,298 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:44,300 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,300 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,300 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:44,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:44,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:44,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:44,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:44,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,302 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:44,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:44,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:44,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:44,303 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:44,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:44,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,309 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:44,309 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:44,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:44,309 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:44,309 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:44,309 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,310 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,310 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,310 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,310 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:44,310 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:44,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:44,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:44,310 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:44,310 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:44,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:44,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:44,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:44,410 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/f24416ec3ede4c68bc924c928292587c 2023-04-27 03:55:44,410 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:44,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:44,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:44,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:44,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:44,415 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:44,415 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:44,415 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:44,415 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:44,415 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,415 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:44,417 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:44,417 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:44,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:44,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,418 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,418 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:44,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:44,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:44,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:44,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:44,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:44,419 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:44,419 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:44,419 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:44,419 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:44,420 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:44,424 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:44,424 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:44,424 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:44,424 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:44,424 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:44,424 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:44,424 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,428 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:44,428 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:44,429 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,429 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:44,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:44,430 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,430 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:44,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:44,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:44,431 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/5e6f8b4eac924e788a1f814e6b3aa776 2023-04-27 03:55:44,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:44,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:44,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33372d53[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:44,433 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:44,433 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,435 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,435 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:44,435 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:44,435 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:44,437 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/5074919511504133a6f3abcf7a731f4f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/5074919511504133a6f3abcf7a731f4f 2023-04-27 03:55:44,438 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,438 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:44,438 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:44,438 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:44,438 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:44,439 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:44,439 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:44,439 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:44,439 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:44,439 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:44,439 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:44,440 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,440 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:44,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,440 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:44,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:44,440 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/5074919511504133a6f3abcf7a731f4f, entries=150, sequenceid=285, filesize=11.9 K 2023-04-27 03:55:44,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:44,441 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:44,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:44,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:44,442 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/f24416ec3ede4c68bc924c928292587c as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/f24416ec3ede4c68bc924c928292587c 2023-04-27 03:55:44,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:44,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:44,444 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,449 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:44,449 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:44,449 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:44,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:44,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:44,449 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:44,449 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:44,449 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,449 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,449 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:44,449 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:44,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:44,449 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:44,449 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:44,450 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:44,450 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/f24416ec3ede4c68bc924c928292587c, entries=150, sequenceid=285, filesize=11.9 K 2023-04-27 03:55:44,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:44,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:44,451 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/5e6f8b4eac924e788a1f814e6b3aa776 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/5e6f8b4eac924e788a1f814e6b3aa776 2023-04-27 03:55:44,456 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/5e6f8b4eac924e788a1f814e6b3aa776, entries=150, sequenceid=285, filesize=11.9 K 2023-04-27 03:55:44,456 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 98cbc3f1a78abce9e99d0349272cd6a4 in 920ms, sequenceid=285, compaction requested=false 2023-04-27 03:55:44,456 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:44,550 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:44,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:44,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:44,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:44,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:44,553 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:44,553 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:44,553 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:44,553 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:44,556 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,556 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:44,557 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:44,557 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:44,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:44,557 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,557 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:44,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:44,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:44,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:44,559 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:44,559 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:44,559 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:44,559 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:44,559 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:44,559 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:44,559 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:44,559 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:44,559 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-27 03:55:44,559 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:44,560 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:44,560 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:44,560 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:44,560 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:44,560 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:44,567 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/032bd2fbeea2434f831bc3d7b65f9a30 2023-04-27 03:55:44,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:44,693 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:44,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:44,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567804765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:44,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567804771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:44,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567804772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:44,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567804873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:44,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567804876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:44,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567804878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:44,978 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/a8723498be7b4d41bb1334b1d3f52d86 2023-04-27 03:55:45,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:45,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567805078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:45,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567805081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:45,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567805083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:45,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567805256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,262 DEBUG [Thread-3254] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4189 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., hostname=jenkins-hbase4.apache.org,40885,1682567618305, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 03:55:45,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:45,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567805272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,278 DEBUG [Thread-3252] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4203 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., hostname=jenkins-hbase4.apache.org,40885,1682567618305, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 03:55:45,388 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/67d0415488f241eda61d42534aaf9980 2023-04-27 03:55:45,392 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/032bd2fbeea2434f831bc3d7b65f9a30 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/032bd2fbeea2434f831bc3d7b65f9a30 2023-04-27 03:55:45,393 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:45,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45914 deadline: 1682567805384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,393 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:45,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567805386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,394 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:45,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567805390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,396 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/032bd2fbeea2434f831bc3d7b65f9a30, entries=150, sequenceid=301, filesize=11.9 K 2023-04-27 03:55:45,396 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/a8723498be7b4d41bb1334b1d3f52d86 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/a8723498be7b4d41bb1334b1d3f52d86 2023-04-27 03:55:45,399 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/a8723498be7b4d41bb1334b1d3f52d86, entries=150, sequenceid=301, filesize=11.9 K 2023-04-27 03:55:45,400 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/67d0415488f241eda61d42534aaf9980 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/67d0415488f241eda61d42534aaf9980 2023-04-27 03:55:45,403 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/67d0415488f241eda61d42534aaf9980, entries=150, sequenceid=301, filesize=11.9 K 2023-04-27 03:55:45,404 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 98cbc3f1a78abce9e99d0349272cd6a4 in 845ms, sequenceid=301, compaction requested=true 2023-04-27 03:55:45,404 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:45,404 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:45,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:45,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:45,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:45,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:45,406 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,406 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:45,407 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:45,407 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:45,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:45,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:45,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:45,409 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:45,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30d8dd97[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:45,409 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:45,410 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:45,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:45,410 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:45,410 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:45,410 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:45,410 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,410 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:45,411 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:45,412 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,412 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:45,412 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:45,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:45,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:45,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:45,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:45,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:45,414 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:45,414 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:45,414 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:45,414 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:45,415 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:45,416 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:45,416 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:45,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:45,416 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:45,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:45,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:45,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:45,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:45,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:45,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:45,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:45,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:45,417 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:45,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:45,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:45,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,419 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:45,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:45,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,422 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,422 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:45,422 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:45,422 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,422 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:45,422 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:45,422 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:45,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:45,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:45,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:45,422 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:45,423 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:45,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:45,423 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:45,423 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:45,423 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:45,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:45,523 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:45,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:45,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:45,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:45,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:45,526 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:45,526 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:45,526 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:45,526 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:45,527 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:45,527 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:45,528 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:45,528 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:45,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:45,528 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,528 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:45,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:45,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:45,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:45,530 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:45,530 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:45,530 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:45,530 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:45,530 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:45,530 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:45,531 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:45,531 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:45,531 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-27 03:55:45,531 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:45,531 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:45,531 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:45,531 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:45,531 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:45,531 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:45,538 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/94701c92c8974cc2be338eab122938c0 2023-04-27 03:55:45,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:45,900 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:45,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:45,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45930 deadline: 1682567805915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:45,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45914 deadline: 1682567805918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:45,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567805919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,950 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/8140ffd2db4b4dab9c145a36583cba17 2023-04-27 03:55:45,963 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/0bb03e5c3a6a4f8089b6cef91148d0cc 2023-04-27 03:55:45,966 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/94701c92c8974cc2be338eab122938c0 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/94701c92c8974cc2be338eab122938c0 2023-04-27 03:55:45,970 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/94701c92c8974cc2be338eab122938c0, entries=150, sequenceid=319, filesize=11.9 K 2023-04-27 03:55:45,971 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/8140ffd2db4b4dab9c145a36583cba17 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/8140ffd2db4b4dab9c145a36583cba17 2023-04-27 03:55:45,974 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/8140ffd2db4b4dab9c145a36583cba17, entries=150, sequenceid=319, filesize=11.9 K 2023-04-27 03:55:45,975 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/0bb03e5c3a6a4f8089b6cef91148d0cc as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/0bb03e5c3a6a4f8089b6cef91148d0cc 2023-04-27 03:55:45,978 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/0bb03e5c3a6a4f8089b6cef91148d0cc, entries=150, sequenceid=319, filesize=11.9 K 2023-04-27 03:55:45,978 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 98cbc3f1a78abce9e99d0349272cd6a4 in 447ms, sequenceid=319, compaction requested=true 2023-04-27 03:55:45,979 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:45,979 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:45,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:45,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:45,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:45,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:45,981 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:45,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:45,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:45,981 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:45,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:45,981 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:45,981 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:45,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:45,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:45,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:45,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@766afe8b[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:45,982 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:45,983 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:45,984 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:45,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:45,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:45,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:45,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:45,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:45,985 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,985 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:45,990 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,990 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:45,990 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:45,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:45,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:45,990 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:45,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:45,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:45,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:45,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:45,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:45,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:45,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:45,993 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:45,993 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:45,993 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:45,993 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:45,993 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:45,994 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:45,994 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:45,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:45,995 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:45,995 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:45,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:45,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:45,998 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:45,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:45,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:45,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:45,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,003 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:46,003 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:46,003 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,003 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,003 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:46,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:46,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:46,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:46,003 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,003 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:46,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:46,003 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,003 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:46,004 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,004 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:46,004 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:46,028 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-27 03:55:46,028 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:46,028 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:46,028 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:46,028 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:46,028 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:46,028 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:46,038 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/71cfff43b2854413ae5f504cdf5154cf 2023-04-27 03:55:46,071 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:46,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45914 deadline: 1682567806064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,071 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:46,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567806064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:46,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45930 deadline: 1682567806065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,104 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:46,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:46,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:46,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:46,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:46,108 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:46,108 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:46,108 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:46,108 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:46,109 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,109 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,116 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:46,116 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:46,116 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,116 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,117 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:46,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:46,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:46,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:46,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:46,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:46,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:46,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:46,121 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:46,121 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:46,121 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:46,121 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:46,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:46,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:46,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:46,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:46,123 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:46,124 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:46,124 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,124 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:46,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:46,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@768fec92[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:46,125 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:46,125 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,126 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,127 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:46,127 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:46,127 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:46,127 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,127 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:46,128 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:46,128 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,128 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:46,128 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:46,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:46,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:46,132 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:46,132 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:46,132 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:46,132 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:46,132 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:46,135 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,135 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:46,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:46,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,135 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:46,135 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:46,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:46,136 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,138 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:46,139 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:46,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:46,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:46,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:46,139 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:46,138 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,139 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,139 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,139 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,139 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,139 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,139 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:46,140 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:46,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:46,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:46,140 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:46,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:46,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:46,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45914 deadline: 1682567806172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:46,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567806172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:46,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45930 deadline: 1682567806178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,240 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:46,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:46,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:46,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:46,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:46,244 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:46,244 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:46,244 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:46,244 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:46,245 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,245 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,246 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:46,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:46,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:46,246 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,247 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,247 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:46,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:46,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:46,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:46,248 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:46,248 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:46,248 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:46,249 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:46,249 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:46,249 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:46,249 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:46,249 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:46,249 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:46,249 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:46,249 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:46,249 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,249 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:46,249 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:46,251 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,251 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:46,251 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,251 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:46,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:46,253 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:46,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c580aa5[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:46,253 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,255 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,255 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:46,255 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:46,256 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:46,256 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,256 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:46,257 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,257 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:46,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:46,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:46,257 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:46,257 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:46,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:46,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:46,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:46,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:46,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:46,260 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:46,260 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:46,260 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:46,260 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:46,260 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:46,261 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,262 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:46,262 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:46,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:46,262 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:46,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,264 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,266 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:46,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:46,266 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:46,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:46,266 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:46,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:46,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:46,266 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,267 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:46,267 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:46,267 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,267 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,267 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,267 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:46,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:46,267 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,267 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,367 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:46,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:46,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:46,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:46,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:46,370 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:46,370 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:46,370 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:46,370 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:46,371 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,371 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,373 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,373 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:46,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:46,374 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,374 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:46,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:46,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:46,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:46,375 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:46,375 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:46,375 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:46,375 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:46,375 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:46,375 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:46,375 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:46,376 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:46,376 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:46,376 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:46,376 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:46,376 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,376 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:46,376 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:46,377 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,377 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:46,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:46,378 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,378 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:46,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:46,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:46,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:46,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:46,380 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:46,380 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a9a44dc[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:46,381 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,381 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:46,381 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:46,382 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:46,382 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,382 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:46,383 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,383 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:46,383 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:46,383 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:46,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:46,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45914 deadline: 1682567806379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:46,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567806379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:46,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:46,386 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:46,386 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:46,386 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:46,386 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:46,386 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:46,391 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,391 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,391 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:46,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:46,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:46,393 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:46,394 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,397 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:46,397 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:46,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:46,397 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:46,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:46,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:46,397 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,397 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,397 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,397 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,397 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:46,397 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:46,397 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:46,397 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,397 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:46,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45930 deadline: 1682567806390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,447 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/9a5f1e88e81f400997176a8c5880f0b3 2023-04-27 03:55:46,498 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:46,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:46,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:46,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:46,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:46,501 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:46,501 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:46,501 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:46,501 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:46,502 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,502 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,506 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:46,506 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:46,506 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,506 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,507 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:46,507 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:46,507 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:46,507 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:46,507 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:46,507 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:46,508 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:46,508 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:46,508 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:46,508 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:46,508 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:46,508 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:46,508 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:46,508 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:46,508 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,508 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:46,508 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:46,513 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,513 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:46,514 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,514 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:46,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:46,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ba57da2[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:46,515 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:46,516 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,517 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,517 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:46,517 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:46,517 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:46,517 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,517 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:46,519 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,519 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:46,519 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:46,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:46,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:46,519 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:46,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:46,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:46,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:46,521 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:46,521 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:46,521 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:46,521 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:46,521 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:46,523 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,524 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:46,524 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,524 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:46,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,525 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:46,531 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:46,531 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:46,531 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:46,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:46,531 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,532 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,532 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,532 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,532 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,532 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:46,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:46,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:46,532 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:46,532 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:46,532 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:46,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:46,632 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:46,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:46,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:46,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:46,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:46,636 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:46,636 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:46,636 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:46,636 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:46,636 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,636 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,638 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,638 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:46,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:46,638 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,638 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:46,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:46,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:46,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:46,639 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:46,639 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:46,639 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:46,639 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:46,639 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:46,640 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:46,640 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:46,640 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:46,640 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:46,640 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:46,640 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:46,640 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,640 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:46,640 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:46,641 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,641 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:46,642 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,642 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:46,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:46,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2247ff38[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:46,644 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:46,644 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,645 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,645 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:46,645 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:46,645 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,645 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:46,645 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:46,648 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,648 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:46,648 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:46,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:46,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:46,648 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:46,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:46,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:46,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:46,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:46,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:46,650 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:46,650 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:46,650 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:46,650 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:46,650 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:46,652 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,652 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:46,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,652 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:46,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:46,653 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:46,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,654 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:46,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,657 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,657 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,657 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:46,657 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:46,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:46,657 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:46,657 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:46,657 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:46,657 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:46,657 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,657 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:46,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:46,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:46,657 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,657 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:46,690 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:46,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45914 deadline: 1682567806684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,690 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:46,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567806686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:46,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45930 deadline: 1682567806700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,757 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:46,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:46,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:46,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:46,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:46,761 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:46,761 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:46,761 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:46,761 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:46,761 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,761 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,763 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,763 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:46,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:46,763 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,763 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:46,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:46,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:46,765 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:46,765 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:46,765 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:46,765 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:46,765 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:46,765 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:46,765 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:46,765 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:46,765 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:46,765 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:46,765 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:46,765 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,765 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:46,765 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:46,768 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,768 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:46,768 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,768 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:46,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:46,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:46,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e76c4f7[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:46,770 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:46,770 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,771 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,771 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,771 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,771 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:46,771 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:46,771 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:46,773 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,773 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:46,773 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:46,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,773 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:46,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:46,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:46,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:46,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:46,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:46,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:46,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:46,775 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:46,775 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:46,775 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:46,775 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:46,775 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:46,777 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,777 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,777 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:46,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:46,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:46,778 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,780 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:46,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,783 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,783 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:46,783 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:46,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:46,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:46,783 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:46,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:46,783 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:46,783 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:46,783 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:46,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:46,783 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,783 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,783 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,783 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:46,784 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:46,857 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/58fd4e32f33d4ceda8cf49d478b2acbf 2023-04-27 03:55:46,861 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/71cfff43b2854413ae5f504cdf5154cf as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/71cfff43b2854413ae5f504cdf5154cf 2023-04-27 03:55:46,864 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/71cfff43b2854413ae5f504cdf5154cf, entries=250, sequenceid=335, filesize=16.7 K 2023-04-27 03:55:46,865 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/9a5f1e88e81f400997176a8c5880f0b3 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/9a5f1e88e81f400997176a8c5880f0b3 2023-04-27 03:55:46,868 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/9a5f1e88e81f400997176a8c5880f0b3, entries=150, sequenceid=335, filesize=11.9 K 2023-04-27 03:55:46,869 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/58fd4e32f33d4ceda8cf49d478b2acbf as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/58fd4e32f33d4ceda8cf49d478b2acbf 2023-04-27 03:55:46,873 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/58fd4e32f33d4ceda8cf49d478b2acbf, entries=150, sequenceid=335, filesize=11.9 K 2023-04-27 03:55:46,874 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 98cbc3f1a78abce9e99d0349272cd6a4 in 846ms, sequenceid=335, compaction requested=true 2023-04-27 03:55:46,874 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:46,874 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:46,874 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:46,874 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 03:55:46,874 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 03:55:46,874 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:46,876 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61340 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 03:55:46,876 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/B is initiating minor compaction (all files) 2023-04-27 03:55:46,876 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/B in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:46,876 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/97c4ba2b20064700801b023518d64de7, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/f24416ec3ede4c68bc924c928292587c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/a8723498be7b4d41bb1334b1d3f52d86, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/8140ffd2db4b4dab9c145a36583cba17, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/9a5f1e88e81f400997176a8c5880f0b3] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=59.9 K 2023-04-27 03:55:46,876 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66220 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 03:55:46,876 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/A is initiating minor compaction (all files) 2023-04-27 03:55:46,876 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/A in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:46,877 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/74a1eac81ab645b1b814c567c790833c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/5074919511504133a6f3abcf7a731f4f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/032bd2fbeea2434f831bc3d7b65f9a30, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/94701c92c8974cc2be338eab122938c0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/71cfff43b2854413ae5f504cdf5154cf] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=64.7 K 2023-04-27 03:55:46,877 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 97c4ba2b20064700801b023518d64de7, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1682567743034 2023-04-27 03:55:46,877 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 74a1eac81ab645b1b814c567c790833c, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1682567743034 2023-04-27 03:55:46,877 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting f24416ec3ede4c68bc924c928292587c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682567743417 2023-04-27 03:55:46,877 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 5074919511504133a6f3abcf7a731f4f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682567743417 2023-04-27 03:55:46,878 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting a8723498be7b4d41bb1334b1d3f52d86, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1682567743554 2023-04-27 03:55:46,878 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 032bd2fbeea2434f831bc3d7b65f9a30, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1682567743554 2023-04-27 03:55:46,878 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 8140ffd2db4b4dab9c145a36583cba17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1682567744764 2023-04-27 03:55:46,878 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 94701c92c8974cc2be338eab122938c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1682567744764 2023-04-27 03:55:46,878 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 9a5f1e88e81f400997176a8c5880f0b3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1682567745909 2023-04-27 03:55:46,878 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 71cfff43b2854413ae5f504cdf5154cf, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1682567745909 2023-04-27 03:55:46,883 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:46,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:46,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:46,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:46,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:46,887 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:46,887 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:46,887 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:46,887 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:46,889 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#B#compaction#467 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:46,890 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#A#compaction#468 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:46,891 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,891 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,892 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:46,892 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:46,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:46,893 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:46,893 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:46,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:46,899 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:46,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:46,899 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:46,899 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:46,899 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:46,902 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:46,902 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:46,902 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:46,902 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:46,902 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-27 03:55:46,902 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:46,902 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:46,902 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:46,902 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:46,903 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:46,903 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:47,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:47,195 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:47,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:47,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45930 deadline: 1682567807227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:47,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567807228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:47,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45914 deadline: 1682567807230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,309 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/2f6644c85d254562bef60e7d7cc07de8 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/2f6644c85d254562bef60e7d7cc07de8 2023-04-27 03:55:47,312 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/666ec23544a14fa08ad1af5b4bed0fed as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/666ec23544a14fa08ad1af5b4bed0fed 2023-04-27 03:55:47,313 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/B of 98cbc3f1a78abce9e99d0349272cd6a4 into 2f6644c85d254562bef60e7d7cc07de8(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:47,313 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:47,313 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/B, priority=11, startTime=1682567746874; duration=0sec 2023-04-27 03:55:47,314 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:47,314 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 03:55:47,316 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61340 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 03:55:47,316 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/C is initiating minor compaction (all files) 2023-04-27 03:55:47,317 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/C in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:47,317 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/882d1ebb8fa049c58e1aa3b44f352f46, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/5e6f8b4eac924e788a1f814e6b3aa776, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/67d0415488f241eda61d42534aaf9980, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/0bb03e5c3a6a4f8089b6cef91148d0cc, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/58fd4e32f33d4ceda8cf49d478b2acbf] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=59.9 K 2023-04-27 03:55:47,317 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 882d1ebb8fa049c58e1aa3b44f352f46, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1682567743034 2023-04-27 03:55:47,317 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 5e6f8b4eac924e788a1f814e6b3aa776, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1682567743417 2023-04-27 03:55:47,318 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 67d0415488f241eda61d42534aaf9980, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1682567743554 2023-04-27 03:55:47,318 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/7f04439000c24e7ca2b276db45f8a294 2023-04-27 03:55:47,318 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/A of 98cbc3f1a78abce9e99d0349272cd6a4 into 666ec23544a14fa08ad1af5b4bed0fed(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:47,318 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:47,318 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/A, priority=11, startTime=1682567746874; duration=0sec 2023-04-27 03:55:47,318 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:47,319 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 0bb03e5c3a6a4f8089b6cef91148d0cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1682567744764 2023-04-27 03:55:47,319 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 58fd4e32f33d4ceda8cf49d478b2acbf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1682567745909 2023-04-27 03:55:47,332 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#C#compaction#471 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:47,340 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/37281145b4e349f78df4435b83f70a71 2023-04-27 03:55:47,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:47,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567807337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:47,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45930 deadline: 1682567807337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:47,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45914 deadline: 1682567807339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:47,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567807545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:47,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45914 deadline: 1682567807545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:47,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45930 deadline: 1682567807547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,753 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/1c769005cfc04fba93c476c6cc8f023e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/1c769005cfc04fba93c476c6cc8f023e 2023-04-27 03:55:47,757 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/C of 98cbc3f1a78abce9e99d0349272cd6a4 into 1c769005cfc04fba93c476c6cc8f023e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:47,758 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:47,758 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/C, priority=11, startTime=1682567746874; duration=0sec 2023-04-27 03:55:47,758 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:47,761 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/d9a98cf7f3124ba2a31d02bcb07edd0e 2023-04-27 03:55:47,764 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/7f04439000c24e7ca2b276db45f8a294 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/7f04439000c24e7ca2b276db45f8a294 2023-04-27 03:55:47,768 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/7f04439000c24e7ca2b276db45f8a294, entries=150, sequenceid=352, filesize=11.9 K 2023-04-27 03:55:47,768 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/37281145b4e349f78df4435b83f70a71 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/37281145b4e349f78df4435b83f70a71 2023-04-27 03:55:47,772 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/37281145b4e349f78df4435b83f70a71, entries=150, sequenceid=352, filesize=11.9 K 2023-04-27 03:55:47,773 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/d9a98cf7f3124ba2a31d02bcb07edd0e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/d9a98cf7f3124ba2a31d02bcb07edd0e 2023-04-27 03:55:47,776 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/d9a98cf7f3124ba2a31d02bcb07edd0e, entries=150, sequenceid=352, filesize=11.9 K 2023-04-27 03:55:47,777 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 98cbc3f1a78abce9e99d0349272cd6a4 in 875ms, sequenceid=352, compaction requested=false 2023-04-27 03:55:47,777 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:47,777 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:47,777 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:47,777 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:47,777 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,777 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:47,777 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:47,779 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,779 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:47,780 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:47,780 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:47,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:47,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:47,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:47,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4612cf41[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:47,782 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:47,783 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:47,784 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:47,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:47,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:47,784 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:47,784 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:47,784 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:47,784 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,784 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:47,787 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:47,787 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:47,787 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,787 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:47,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:47,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:47,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:47,791 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:47,791 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:47,791 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:47,791 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:47,791 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:47,793 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:47,793 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:47,793 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:47,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:47,793 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:47,795 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,797 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,797 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:47,797 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:47,797 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,797 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:47,797 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:47,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:47,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:47,797 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:47,798 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:47,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:47,797 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:47,798 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:47,798 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:47,798 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:47,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:47,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:47,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:47,855 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-27 03:55:47,855 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:47,855 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:47,855 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:47,855 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:47,855 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:47,855 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:47,862 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/936a3440000c44cebc29794c077a764d 2023-04-27 03:55:47,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:47,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45930 deadline: 1682567807877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:47,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45914 deadline: 1682567807878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:47,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567807880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,898 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:47,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:47,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:47,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:47,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:47,901 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:47,901 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:47,901 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:47,901 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:47,901 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:47,901 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:47,903 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:47,903 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:47,903 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:47,903 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:47,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:47,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:47,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:47,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:47,904 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:47,904 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:47,904 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:47,904 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:47,904 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:47,904 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:47,904 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:47,905 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:47,905 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:47,905 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:47,905 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:47,905 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,905 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:47,905 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:47,907 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,907 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:47,907 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:47,907 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:47,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:47,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46dae24c[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:47,909 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:47,909 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:47,910 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:47,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:47,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:47,910 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:47,911 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:47,911 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,911 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:47,911 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:47,912 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,912 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:47,912 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:47,912 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:47,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:47,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:47,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:47,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:47,915 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:47,915 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:47,915 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:47,915 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:47,915 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:47,917 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:47,917 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:47,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:47,917 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:47,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:47,917 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:47,918 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:47,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,922 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,922 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:47,922 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:47,922 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,922 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:47,922 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:47,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:47,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:47,922 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:47,922 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:47,922 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:47,922 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:47,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:47,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:47,923 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:47,923 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:47,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:47,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:47,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:47,982 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:47,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45930 deadline: 1682567807981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:47,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45914 deadline: 1682567807984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:47,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:47,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567807985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,023 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:48,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:48,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:48,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:48,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:48,026 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:48,026 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:48,026 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:48,026 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:48,026 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,026 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,027 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:48,028 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:48,028 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,028 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:48,029 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:48,029 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:48,029 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:48,029 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:48,029 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:48,029 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:48,029 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:48,029 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:48,029 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:48,029 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:48,029 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:48,029 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:48,029 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:48,029 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,029 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:48,029 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:48,031 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,031 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:48,031 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:48,031 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:48,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:48,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bf97b90[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:48,033 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:48,033 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,034 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,034 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:48,034 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:48,034 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,034 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:48,034 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:48,036 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:48,036 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:48,036 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:48,036 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:48,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:48,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:48,038 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:48,038 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:48,038 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:48,038 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:48,039 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:48,040 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,040 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:48,040 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:48,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:48,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,041 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:48,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,042 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:48,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,044 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,044 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,044 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,044 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,044 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,044 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:48,044 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:48,044 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:48,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:48,044 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:48,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:48,044 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:48,044 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:48,044 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:48,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:48,144 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:48,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:48,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:48,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:48,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:48,147 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:48,147 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:48,147 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:48,147 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:48,148 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,148 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,149 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,149 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:48,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:48,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:48,149 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,149 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:48,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:48,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:48,150 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:48,150 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:48,150 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:48,150 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:48,150 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:48,150 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:48,150 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:48,151 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:48,151 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:48,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:48,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:48,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:48,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:48,153 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,153 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:48,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:48,154 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,154 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:48,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:48,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:48,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:48,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:48,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73805d64[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:48,155 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:48,155 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,156 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,156 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:48,156 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:48,156 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,156 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:48,156 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:48,160 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,161 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:48,161 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:48,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:48,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:48,161 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:48,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:48,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:48,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:48,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:48,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:48,163 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:48,163 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:48,163 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:48,163 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:48,163 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:48,165 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,165 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,165 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:48,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:48,166 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:48,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,167 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:48,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,169 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,169 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:48,169 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:48,169 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,169 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:48,169 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:48,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:48,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:48,169 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:48,169 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,169 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,170 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,169 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:48,170 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:48,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:48,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45930 deadline: 1682567808183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:48,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45914 deadline: 1682567808190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:48,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567808190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,270 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:48,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:48,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:48,273 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/0ee4887afff847c0bb6cf32285316422 2023-04-27 03:55:48,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:48,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:48,273 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:48,273 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:48,273 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:48,273 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:48,274 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,274 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,276 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:48,276 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:48,276 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,276 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:48,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:48,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:48,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:48,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:48,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:48,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:48,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:48,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:48,278 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:48,278 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:48,278 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:48,278 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:48,280 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:48,280 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:48,280 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,280 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:48,280 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:48,281 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,281 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:48,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:48,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:48,282 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,282 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:48,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:48,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:48,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:48,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2338fc92[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:48,284 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:48,284 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,286 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,286 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:48,286 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:48,286 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:48,286 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,286 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:48,288 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:48,288 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:48,288 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:48,288 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:48,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:48,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:48,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:48,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:48,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:48,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:48,290 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:48,290 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:48,290 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:48,290 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:48,290 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:48,292 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,292 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:48,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:48,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,293 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:48,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:48,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:48,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:48,293 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:48,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,295 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,296 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,296 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,296 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,297 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,297 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,297 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,297 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,297 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:48,296 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:48,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:48,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:48,297 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:48,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:48,297 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:48,297 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:48,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:48,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:48,397 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:48,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:48,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:48,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:48,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:48,403 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:48,404 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:48,404 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:48,404 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:48,404 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,404 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,407 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,407 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:48,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:48,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:48,407 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,407 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:48,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:48,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:48,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:48,408 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:48,408 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:48,408 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:48,408 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:48,408 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:48,408 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:48,409 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:48,409 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:48,409 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:48,409 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:48,409 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:48,409 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:48,409 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,409 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:48,409 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:48,412 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,412 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:48,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:48,413 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,413 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:48,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:48,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:48,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:48,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:48,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32e1a23b[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:48,414 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:48,415 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,416 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,416 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,416 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:48,416 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:48,416 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:48,416 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:48,418 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:48,418 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:48,418 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,418 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:48,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:48,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:48,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:48,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:48,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:48,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:48,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:48,420 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:48,420 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:48,420 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:48,420 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:48,420 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:48,423 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,423 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:48,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:48,423 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:48,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:48,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:48,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:48,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,424 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:48,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,425 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:48,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,427 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,427 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,427 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,427 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,427 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,427 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:48,427 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:48,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:48,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:48,427 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:48,427 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,427 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:48,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:48,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:48,428 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:48,428 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:48,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:48,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45930 deadline: 1682567808492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,502 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:48,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45914 deadline: 1682567808498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:48,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567808499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,528 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:48,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:48,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:48,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:48,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:48,531 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:48,531 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:48,531 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:48,531 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:48,532 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,532 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,533 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,533 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:48,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:48,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:48,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,534 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,534 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:48,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:48,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,534 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:48,535 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:48,535 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:48,535 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:48,535 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:48,535 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:48,535 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:48,535 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:48,535 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:48,535 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:48,535 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:48,535 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:48,535 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:48,536 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:48,536 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,536 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:48,536 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:48,537 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,537 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:48,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:48,538 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,538 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:48,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:48,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:48,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:48,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:48,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22546a0c[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:48,539 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:48,539 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,541 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,542 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:48,542 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:48,542 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:48,542 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,542 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:48,543 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:48,543 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:48,543 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:48,543 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:48,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:48,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:48,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:48,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:48,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:48,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:48,546 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:48,546 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:48,546 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:48,546 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:48,546 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:48,550 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,550 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,550 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:48,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,550 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,552 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:48,555 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,555 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,555 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:48,555 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:48,555 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:48,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:48,555 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,555 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,555 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,555 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:48,555 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:48,556 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:48,556 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:48,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:48,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:48,656 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:48,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:48,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:48,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:48,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:48,659 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:48,659 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:48,659 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:48,659 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:48,660 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,660 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,661 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:48,661 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:48,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:48,661 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,662 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,662 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:48,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:48,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:48,663 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:48,663 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:48,663 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:48,663 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:48,663 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:48,663 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:48,663 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:48,663 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:48,663 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:48,663 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:48,663 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:48,663 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,663 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:48,663 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:48,665 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:48,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:48,665 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:48,666 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,666 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:48,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:48,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@534e785f[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:48,667 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:48,667 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,668 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,668 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:48,668 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:48,668 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:48,668 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,668 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:48,670 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:48,670 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:48,670 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,670 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:48,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:48,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:48,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:48,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:48,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:48,673 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:48,673 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:48,673 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:48,673 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:48,673 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:48,675 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,675 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:48,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,675 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:48,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,676 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:48,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,677 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,679 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,679 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,679 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,679 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,679 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,679 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:48,679 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,679 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:48,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:48,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:48,679 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:48,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:48,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:48,679 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:48,680 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:48,680 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:48,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:48,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:48,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:48,684 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/cbbda517f96e4bd684a943f381c1882c 2023-04-27 03:55:48,688 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/936a3440000c44cebc29794c077a764d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/936a3440000c44cebc29794c077a764d 2023-04-27 03:55:48,691 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/936a3440000c44cebc29794c077a764d, entries=200, sequenceid=371, filesize=14.3 K 2023-04-27 03:55:48,692 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/0ee4887afff847c0bb6cf32285316422 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/0ee4887afff847c0bb6cf32285316422 2023-04-27 03:55:48,696 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/0ee4887afff847c0bb6cf32285316422, entries=150, sequenceid=371, filesize=11.9 K 2023-04-27 03:55:48,696 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/cbbda517f96e4bd684a943f381c1882c as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/cbbda517f96e4bd684a943f381c1882c 2023-04-27 03:55:48,702 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/cbbda517f96e4bd684a943f381c1882c, entries=150, sequenceid=371, filesize=11.9 K 2023-04-27 03:55:48,702 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 98cbc3f1a78abce9e99d0349272cd6a4 in 847ms, sequenceid=371, compaction requested=true 2023-04-27 03:55:48,702 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:48,703 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:48,703 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:48,703 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:48,703 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:48,703 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:48,703 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:48,704 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/B is initiating minor compaction (all files) 2023-04-27 03:55:48,704 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39578 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:48,704 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/B in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:48,704 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/A is initiating minor compaction (all files) 2023-04-27 03:55:48,704 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/2f6644c85d254562bef60e7d7cc07de8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/37281145b4e349f78df4435b83f70a71, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/0ee4887afff847c0bb6cf32285316422] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=36.3 K 2023-04-27 03:55:48,704 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/A in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:48,704 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/666ec23544a14fa08ad1af5b4bed0fed, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/7f04439000c24e7ca2b276db45f8a294, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/936a3440000c44cebc29794c077a764d] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=38.7 K 2023-04-27 03:55:48,704 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 2f6644c85d254562bef60e7d7cc07de8, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1682567745909 2023-04-27 03:55:48,704 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 666ec23544a14fa08ad1af5b4bed0fed, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1682567745909 2023-04-27 03:55:48,704 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 37281145b4e349f78df4435b83f70a71, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1682567746040 2023-04-27 03:55:48,705 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 7f04439000c24e7ca2b276db45f8a294, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1682567746040 2023-04-27 03:55:48,705 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 0ee4887afff847c0bb6cf32285316422, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1682567747227 2023-04-27 03:55:48,705 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 936a3440000c44cebc29794c077a764d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1682567747226 2023-04-27 03:55:48,710 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#B#compaction#476 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:48,711 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#A#compaction#477 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:48,780 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:48,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:48,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:48,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:48,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:48,783 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:48,783 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:48,783 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:48,783 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:48,783 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,783 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,784 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,784 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:48,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:48,785 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:48,785 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:48,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:48,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:48,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:48,786 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:48,786 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:48,786 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:48,786 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:48,786 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:48,786 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:48,786 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:48,786 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-27 03:55:48,786 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:48,786 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:48,786 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:48,786 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:48,787 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:48,787 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:48,799 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/516f58a0dd554c848a2831fa4c638838 2023-04-27 03:55:48,817 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/4cf3969ff9214f5b85ac6b7cbc82cd57 2023-04-27 03:55:49,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:49,005 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:49,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:49,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45914 deadline: 1682567809030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:49,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567809030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:49,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45930 deadline: 1682567809033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,130 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/c0cdd5700eca4a0e8841b3e32c85b5b5 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/c0cdd5700eca4a0e8841b3e32c85b5b5 2023-04-27 03:55:49,133 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/B of 98cbc3f1a78abce9e99d0349272cd6a4 into c0cdd5700eca4a0e8841b3e32c85b5b5(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:49,133 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:49,133 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/B, priority=13, startTime=1682567748703; duration=0sec 2023-04-27 03:55:49,134 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:49,134 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:49,137 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:49,137 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/C is initiating minor compaction (all files) 2023-04-27 03:55:49,137 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/C in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:49,137 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/1c769005cfc04fba93c476c6cc8f023e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/d9a98cf7f3124ba2a31d02bcb07edd0e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/cbbda517f96e4bd684a943f381c1882c] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=36.3 K 2023-04-27 03:55:49,138 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 1c769005cfc04fba93c476c6cc8f023e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1682567745909 2023-04-27 03:55:49,139 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting d9a98cf7f3124ba2a31d02bcb07edd0e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1682567746040 2023-04-27 03:55:49,139 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting cbbda517f96e4bd684a943f381c1882c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1682567747227 2023-04-27 03:55:49,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:49,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45914 deadline: 1682567809143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:49,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567809143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:49,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45930 deadline: 1682567809145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,147 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/bee61f531fb0439aa9ff7a6f14b443b5 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/bee61f531fb0439aa9ff7a6f14b443b5 2023-04-27 03:55:49,158 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/A of 98cbc3f1a78abce9e99d0349272cd6a4 into bee61f531fb0439aa9ff7a6f14b443b5(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:49,158 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:49,158 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/A, priority=13, startTime=1682567748703; duration=0sec 2023-04-27 03:55:49,158 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:49,158 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#C#compaction#481 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:49,228 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/b9361a27edbe4cb18fac13cd54b197c5 2023-04-27 03:55:49,232 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/516f58a0dd554c848a2831fa4c638838 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/516f58a0dd554c848a2831fa4c638838 2023-04-27 03:55:49,236 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/516f58a0dd554c848a2831fa4c638838, entries=150, sequenceid=387, filesize=11.9 K 2023-04-27 03:55:49,237 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/4cf3969ff9214f5b85ac6b7cbc82cd57 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/4cf3969ff9214f5b85ac6b7cbc82cd57 2023-04-27 03:55:49,240 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/4cf3969ff9214f5b85ac6b7cbc82cd57, entries=150, sequenceid=387, filesize=11.9 K 2023-04-27 03:55:49,241 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/b9361a27edbe4cb18fac13cd54b197c5 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/b9361a27edbe4cb18fac13cd54b197c5 2023-04-27 03:55:49,244 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/b9361a27edbe4cb18fac13cd54b197c5, entries=150, sequenceid=387, filesize=11.9 K 2023-04-27 03:55:49,245 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 98cbc3f1a78abce9e99d0349272cd6a4 in 459ms, sequenceid=387, compaction requested=false 2023-04-27 03:55:49,245 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:49,245 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:49,245 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:49,245 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:49,245 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,245 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:49,245 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:49,248 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,248 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:49,248 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,248 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:49,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:49,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:49,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:49,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37f6c2c8[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:49,249 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:49,249 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,251 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,251 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:49,251 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:49,251 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,251 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:49,251 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:49,252 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:49,252 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,252 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:49,252 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:49,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:49,255 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:49,255 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:49,255 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:49,255 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:49,255 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:49,256 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,256 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,257 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:49,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:49,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:49,257 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:49,258 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,261 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,261 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:49,261 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:49,261 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:49,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:49,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:49,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:49,261 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:49,261 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:49,262 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,262 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,262 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,262 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,262 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:49,262 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:49,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:49,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:49,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:49,284 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-27 03:55:49,284 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:49,284 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:49,284 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:49,284 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:49,284 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:49,284 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:49,291 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/0cb7409bfd344f2a932629ebdc135069 2023-04-27 03:55:49,301 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/7dc9ce56c32d42e686ecd4a347033503 2023-04-27 03:55:49,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:49,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567809336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:49,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567809337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:49,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45930 deadline: 1682567809347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:49,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45914 deadline: 1682567809348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:49,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567809348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,362 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:49,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:49,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:49,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:49,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:49,365 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:49,365 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:49,365 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:49,365 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:49,366 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,366 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:49,367 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:49,367 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:49,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:49,367 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,368 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:49,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:49,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:49,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:49,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:49,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:49,370 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:49,370 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:49,370 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:49,370 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:49,370 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:49,370 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:49,371 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:49,371 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:49,371 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:49,371 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:49,371 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:49,371 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,371 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:49,371 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:49,372 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,372 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:49,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:49,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:49,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:49,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:49,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:49,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8b94e77[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:49,374 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:49,375 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,376 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,376 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:49,376 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:49,376 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:49,376 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,376 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:49,379 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,379 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:49,379 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:49,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:49,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:49,379 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:49,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:49,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:49,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:49,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:49,381 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:49,381 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:49,381 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:49,381 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:49,381 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:49,383 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,383 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:49,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:49,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,383 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:49,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:49,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:49,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:49,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,384 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:49,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:49,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:49,385 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,391 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,391 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:49,391 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:49,391 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,391 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:49,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:49,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:49,391 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,391 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,391 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,391 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:49,391 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:49,391 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:49,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:49,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:49,391 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:49,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:49,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:49,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:49,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:49,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567809442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:49,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567809443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,492 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:49,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:49,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:49,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:49,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:49,495 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:49,495 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:49,495 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:49,495 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:49,496 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,496 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:49,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:49,497 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:49,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:49,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:49,498 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,498 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:49,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:49,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:49,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:49,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:49,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:49,499 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:49,499 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:49,499 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:49,499 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:49,499 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:49,499 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:49,499 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:49,499 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:49,499 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:49,499 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:49,499 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:49,499 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,499 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:49,499 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:49,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,501 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:49,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:49,501 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,502 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:49,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:49,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:49,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:49,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:49,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:49,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@93c5166[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:49,504 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:49,504 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,506 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,506 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:49,507 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:49,507 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:49,507 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,507 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:49,509 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:49,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:49,509 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:49,509 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:49,509 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:49,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:49,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:49,513 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:49,513 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:49,513 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:49,513 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:49,513 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:49,514 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:49,514 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,514 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:49,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:49,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:49,515 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,517 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,519 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:49,519 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,519 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:49,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:49,519 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:49,519 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:49,519 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,519 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,519 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:49,519 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:49,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:49,519 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,520 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:49,520 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:49,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:49,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:49,570 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/e5b2fabf84b843a59861a8463acf332d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/e5b2fabf84b843a59861a8463acf332d 2023-04-27 03:55:49,574 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/C of 98cbc3f1a78abce9e99d0349272cd6a4 into e5b2fabf84b843a59861a8463acf332d(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:49,574 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:49,574 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/C, priority=13, startTime=1682567748703; duration=0sec 2023-04-27 03:55:49,575 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:49,620 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:49,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:49,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:49,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:49,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:49,623 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:49,623 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:49,624 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:49,624 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:49,624 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,624 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:49,626 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,626 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:49,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:49,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:49,626 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,626 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:49,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:49,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:49,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:49,628 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:49,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:49,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:49,629 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:49,629 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:49,629 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:49,630 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:49,630 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:49,630 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:49,630 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:49,630 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:49,630 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:49,630 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:49,630 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:49,630 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,630 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:49,630 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:49,632 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,632 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:49,632 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,632 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:49,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:49,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29905c9e[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:49,634 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:49,634 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,635 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,635 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:49,635 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:49,635 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:49,635 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,635 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:49,637 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,638 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:49,638 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:49,638 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:49,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:49,639 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:49,639 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:49,639 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:49,639 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:49,639 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:49,642 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,642 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:49,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,642 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:49,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:49,643 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:49,644 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,646 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,647 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:49,647 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:49,647 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,647 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,647 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:49,647 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:49,647 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:49,647 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:49,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:49,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:49,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:49,647 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:49,647 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:49,647 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:49,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:49,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:49,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567809646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:49,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45930 deadline: 1682567809651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:49,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567809651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:49,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45914 deadline: 1682567809652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:49,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567809655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,709 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/513df438a72649538dbfee7f48d8c5df 2023-04-27 03:55:49,713 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/0cb7409bfd344f2a932629ebdc135069 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/0cb7409bfd344f2a932629ebdc135069 2023-04-27 03:55:49,716 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/0cb7409bfd344f2a932629ebdc135069, entries=200, sequenceid=406, filesize=14.3 K 2023-04-27 03:55:49,717 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/7dc9ce56c32d42e686ecd4a347033503 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/7dc9ce56c32d42e686ecd4a347033503 2023-04-27 03:55:49,720 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/7dc9ce56c32d42e686ecd4a347033503, entries=150, sequenceid=406, filesize=11.9 K 2023-04-27 03:55:49,721 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/513df438a72649538dbfee7f48d8c5df as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/513df438a72649538dbfee7f48d8c5df 2023-04-27 03:55:49,724 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/513df438a72649538dbfee7f48d8c5df, entries=150, sequenceid=406, filesize=11.9 K 2023-04-27 03:55:49,726 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 98cbc3f1a78abce9e99d0349272cd6a4 in 441ms, sequenceid=406, compaction requested=true 2023-04-27 03:55:49,726 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:49,726 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:49,726 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:49,726 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:49,726 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:49,726 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:49,727 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39680 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:49,727 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:49,727 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/A is initiating minor compaction (all files) 2023-04-27 03:55:49,727 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/B is initiating minor compaction (all files) 2023-04-27 03:55:49,727 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/A in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:49,727 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/B in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:49,727 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/bee61f531fb0439aa9ff7a6f14b443b5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/516f58a0dd554c848a2831fa4c638838, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/0cb7409bfd344f2a932629ebdc135069] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=38.8 K 2023-04-27 03:55:49,727 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/c0cdd5700eca4a0e8841b3e32c85b5b5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/4cf3969ff9214f5b85ac6b7cbc82cd57, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/7dc9ce56c32d42e686ecd4a347033503] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=36.4 K 2023-04-27 03:55:49,728 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting bee61f531fb0439aa9ff7a6f14b443b5, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1682567747227 2023-04-27 03:55:49,728 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting c0cdd5700eca4a0e8841b3e32c85b5b5, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1682567747227 2023-04-27 03:55:49,728 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 516f58a0dd554c848a2831fa4c638838, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1682567747876 2023-04-27 03:55:49,728 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 4cf3969ff9214f5b85ac6b7cbc82cd57, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1682567747876 2023-04-27 03:55:49,728 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 0cb7409bfd344f2a932629ebdc135069, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1682567749018 2023-04-27 03:55:49,728 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 7dc9ce56c32d42e686ecd4a347033503, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1682567749018 2023-04-27 03:55:49,734 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#B#compaction#485 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:49,735 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#A#compaction#486 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:49,747 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:49,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:49,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:49,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:49,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:49,751 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:49,751 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:49,751 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:49,751 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:49,752 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,752 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:49,753 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:49,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:49,753 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:49,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:49,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:49,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:49,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:49,754 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:49,755 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:49,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:49,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:49,755 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:49,755 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:49,755 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:49,756 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:49,756 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:49,756 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:49,756 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:49,756 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-27 03:55:49,756 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:49,756 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:49,756 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:49,756 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:49,756 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:49,756 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:49,767 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=422 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/2b8f6d4d445542bd80a39ba5ebf732b1 2023-04-27 03:55:49,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:49,957 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:50,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:50,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567810002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:50,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567810009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:50,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567810113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:50,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567810120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,152 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/b2fd8fa7114c4078ac9dcfbf2aa48f1a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/b2fd8fa7114c4078ac9dcfbf2aa48f1a 2023-04-27 03:55:50,156 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/B of 98cbc3f1a78abce9e99d0349272cd6a4 into b2fd8fa7114c4078ac9dcfbf2aa48f1a(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:50,156 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:50,156 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/B, priority=13, startTime=1682567749726; duration=0sec 2023-04-27 03:55:50,157 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:50,157 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:50,158 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:50,158 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/C is initiating minor compaction (all files) 2023-04-27 03:55:50,158 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/C in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:50,158 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/e5b2fabf84b843a59861a8463acf332d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/b9361a27edbe4cb18fac13cd54b197c5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/513df438a72649538dbfee7f48d8c5df] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=36.4 K 2023-04-27 03:55:50,158 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting e5b2fabf84b843a59861a8463acf332d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1682567747227 2023-04-27 03:55:50,159 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting b9361a27edbe4cb18fac13cd54b197c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1682567747876 2023-04-27 03:55:50,159 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/4261f2d032d3455db501d51c3d7deeff as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/4261f2d032d3455db501d51c3d7deeff 2023-04-27 03:55:50,159 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 513df438a72649538dbfee7f48d8c5df, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1682567749018 2023-04-27 03:55:50,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:50,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45930 deadline: 1682567810157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,163 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/A of 98cbc3f1a78abce9e99d0349272cd6a4 into 4261f2d032d3455db501d51c3d7deeff(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:50,163 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:50,163 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/A, priority=13, startTime=1682567749726; duration=0sec 2023-04-27 03:55:50,163 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:50,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:50,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567810160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:50,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45914 deadline: 1682567810160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,166 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#C#compaction#489 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:50,183 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=422 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/c30501fbfdb24fdf90f05b35d070c82d 2023-04-27 03:55:50,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:50,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567810317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:50,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567810327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,574 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/100e85c16d67411d8fe30233be9c5931 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/100e85c16d67411d8fe30233be9c5931 2023-04-27 03:55:50,578 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/C of 98cbc3f1a78abce9e99d0349272cd6a4 into 100e85c16d67411d8fe30233be9c5931(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:50,578 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:50,578 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/C, priority=13, startTime=1682567749726; duration=0sec 2023-04-27 03:55:50,578 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:50,592 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=422 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/3323f6a7c03c4d808dfd97ef2a4ebff2 2023-04-27 03:55:50,596 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/2b8f6d4d445542bd80a39ba5ebf732b1 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/2b8f6d4d445542bd80a39ba5ebf732b1 2023-04-27 03:55:50,600 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/2b8f6d4d445542bd80a39ba5ebf732b1, entries=150, sequenceid=422, filesize=11.9 K 2023-04-27 03:55:50,600 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.StoreScanner(953): StoreScanner already has the close lock. There is no need to updateReaders 2023-04-27 03:55:50,601 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/c30501fbfdb24fdf90f05b35d070c82d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/c30501fbfdb24fdf90f05b35d070c82d 2023-04-27 03:55:50,604 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/c30501fbfdb24fdf90f05b35d070c82d, entries=150, sequenceid=422, filesize=11.9 K 2023-04-27 03:55:50,604 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/3323f6a7c03c4d808dfd97ef2a4ebff2 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/3323f6a7c03c4d808dfd97ef2a4ebff2 2023-04-27 03:55:50,607 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/3323f6a7c03c4d808dfd97ef2a4ebff2, entries=150, sequenceid=422, filesize=11.9 K 2023-04-27 03:55:50,608 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 98cbc3f1a78abce9e99d0349272cd6a4 in 852ms, sequenceid=422, compaction requested=false 2023-04-27 03:55:50,608 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:50,608 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:50,608 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:50,608 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:50,608 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,608 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:50,608 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:50,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:50,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:50,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:50,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28f13be6[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:50,613 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:50,614 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,616 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:50,616 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:50,616 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:50,616 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,616 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:50,617 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:50,617 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:50,617 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,617 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:50,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:50,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:50,620 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:50,620 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:50,620 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:50,620 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:50,620 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:50,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:50,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:50,622 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-27 03:55:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:50,622 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,622 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:50,622 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:50,623 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:50,623 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:50,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:50,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:50,623 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:50,623 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:50,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:50,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:50,624 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:50,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:50,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:50,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,626 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:50,626 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:50,626 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,627 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:50,627 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:50,627 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,627 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,627 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,627 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:50,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:50,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:50,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:50,627 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:50,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:50,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:50,627 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:50,627 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:50,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:50,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:50,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567810659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:50,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567810665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,727 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:50,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:50,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:50,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:50,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:50,731 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:50,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:50,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:50,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:50,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,732 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:50,737 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:50,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:50,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:50,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:50,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:50,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:50,738 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:50,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:50,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:50,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:50,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:50,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:50,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:50,739 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:50,739 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:50,739 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:50,739 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:50,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:50,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:50,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:50,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:50,740 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,740 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:50,741 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:50,741 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:50,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:50,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:50,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a791279[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:50,744 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:50,744 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,745 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,745 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:50,745 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:50,745 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,745 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:50,745 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:50,748 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:50,748 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:50,748 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:50,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:50,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:50,752 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:50,752 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:50,752 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:50,752 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:50,752 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:50,753 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,753 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:50,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,753 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:50,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,755 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:50,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:50,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,756 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:50,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:50,758 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:50,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:50,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:50,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:50,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:50,758 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:50,759 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:50,759 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,759 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:50,759 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:50,759 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:50,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567810766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:50,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567810768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,859 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:50,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:50,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:50,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:50,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:50,863 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:50,863 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:50,863 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:50,863 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:50,863 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,863 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:50,864 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:50,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:50,865 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,865 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:50,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:50,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:50,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:50,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:50,866 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:50,866 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:50,866 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:50,866 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:50,866 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:50,866 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:50,866 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:50,866 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:50,867 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:50,867 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:50,867 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:50,867 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,867 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:50,867 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:50,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,870 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:50,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:50,870 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,870 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:50,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:50,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:50,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:50,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:50,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:50,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@125d931[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:50,872 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:50,872 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,873 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,873 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:50,873 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,874 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:50,874 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:50,874 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:50,875 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:50,875 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:50,875 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,875 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:50,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:50,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:50,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:50,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:50,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:50,878 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:50,878 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:50,878 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:50,878 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:50,878 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:50,880 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,880 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:50,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,880 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:50,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:50,880 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:50,881 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,885 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,885 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:50,885 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:50,885 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,885 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,885 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:50,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:50,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:50,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:50,885 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:50,885 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:50,885 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,885 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,886 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:50,886 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:50,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:50,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:50,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:50,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567810969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:50,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567810975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,986 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:50,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:50,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:50,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:50,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:50,989 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:50,989 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:50,989 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:50,989 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:50,989 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,989 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:50,991 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:50,991 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:50,991 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,991 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:50,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:50,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:50,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:50,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:50,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:50,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:50,992 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:50,992 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:50,992 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:50,992 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:50,993 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:50,993 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:50,993 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:50,993 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:50,993 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:50,993 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:50,993 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:50,993 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,993 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:50,993 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:50,994 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,994 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:50,995 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,995 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:50,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:50,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2afa0127[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:50,996 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:50,996 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,997 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:50,998 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:50,998 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:50,998 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:50,998 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:50,998 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:51,002 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:51,002 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:51,002 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,002 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:51,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:51,004 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:51,004 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:51,005 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:51,005 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:51,005 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:51,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:51,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:51,007 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,007 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:51,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,007 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:51,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:51,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:51,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:51,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,008 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,010 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:51,010 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,010 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:51,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:51,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:51,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:51,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:51,010 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:51,010 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,011 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,011 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,011 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:51,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:51,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:51,011 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,011 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:51,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:51,011 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,011 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,031 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/4876430557d04dec933a303f8b283248 2023-04-27 03:55:51,111 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:51,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:51,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:51,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:51,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:51,114 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:51,115 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:51,115 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:51,115 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:51,115 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,115 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,116 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:51,116 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:51,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,117 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:51,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:51,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:51,118 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:51,118 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:51,118 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:51,118 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:51,118 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:51,118 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:51,118 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:51,118 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:51,118 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:51,118 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:51,118 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:51,119 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,119 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:51,119 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:51,123 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,123 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:51,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:51,125 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,125 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:51,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:51,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:51,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:51,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:51,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bf53195[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:51,126 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:51,126 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,127 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,127 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,127 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:51,127 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:51,127 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:51,127 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:51,129 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:51,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:51,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:51,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:51,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:51,131 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:51,131 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:51,131 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:51,131 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:51,132 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:51,138 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,138 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:51,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:51,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,138 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:51,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:51,139 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:51,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:51,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:51,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,140 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:51,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,142 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,142 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,142 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,142 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,142 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:51,142 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:51,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:51,142 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:51,142 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:51,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:51,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:51,142 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:51,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:51,142 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:51,142 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,143 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:51,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:51,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:51,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567811167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:51,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45930 deadline: 1682567811170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,180 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:51,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45914 deadline: 1682567811175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,243 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:51,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:51,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:51,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:51,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:51,246 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:51,246 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:51,246 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:51,246 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:51,246 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,246 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,247 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:51,247 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:51,248 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,248 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:51,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:51,250 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:51,250 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:51,250 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:51,250 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:51,250 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:51,250 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:51,250 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:51,250 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:51,250 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:51,250 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:51,250 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:51,250 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,250 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:51,250 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:51,252 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,252 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:51,252 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,252 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:51,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:51,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@293d1048[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:51,253 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:51,253 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,255 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,255 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:51,255 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:51,255 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:51,255 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,256 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:51,256 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,256 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:51,257 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:51,257 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:51,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:51,259 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:51,259 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:51,259 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:51,259 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:51,259 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:51,260 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,260 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,260 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:51,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:51,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:51,261 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,262 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:51,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,264 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,264 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,264 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:51,264 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:51,264 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:51,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:51,264 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:51,264 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:51,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:51,264 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,265 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,265 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,265 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:51,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:51,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:51,265 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:51,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:51,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45982 deadline: 1682567811276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:51,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567811280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,365 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:51,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:51,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:51,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:51,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:51,368 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:51,368 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:51,368 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:51,368 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:51,369 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,369 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,371 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:51,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:51,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:51,371 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,372 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,372 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:51,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:51,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:51,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:51,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:51,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:51,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:51,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:51,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:51,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:51,373 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:51,373 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:51,373 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:51,373 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:51,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:51,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:51,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:51,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:51,375 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:51,375 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:51,376 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,376 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:51,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:51,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6feb80f[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:51,377 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:51,377 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,378 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,378 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:51,378 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:51,378 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:51,378 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,378 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:51,380 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:51,380 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:51,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,380 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:51,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:51,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:51,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:51,382 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:51,382 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:51,382 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:51,382 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:51,382 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:51,385 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,385 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,385 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:51,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:51,385 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:51,386 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,390 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,390 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,390 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,390 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,390 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,390 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,391 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:51,391 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:51,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:51,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:51,390 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:51,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:51,391 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:51,391 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:51,391 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:51,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:51,441 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/058cea19791749e6b7a1978d05ac0f2d 2023-04-27 03:55:51,491 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:51,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:51,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:51,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:51,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:51,495 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:51,495 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:51,495 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:51,495 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:51,495 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,495 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,497 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:51,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:51,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:51,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,498 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,498 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:51,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:51,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:51,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:51,498 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:51,498 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:51,498 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:51,498 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:51,498 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:51,498 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:51,498 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:51,499 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:51,499 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:51,500 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,500 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:51,501 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,501 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:51,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:51,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c7762ce[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:51,502 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:51,502 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,505 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,505 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:51,505 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:51,505 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:51,505 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,505 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:51,509 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:51,509 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:51,509 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,509 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:51,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:51,512 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:51,512 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:51,512 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:51,512 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:51,512 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:51,513 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,513 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:51,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,513 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:51,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:51,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:51,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:51,514 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:51,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:51,515 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:51,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,517 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,517 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,517 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,517 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:51,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:51,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:51,517 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,517 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:51,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:51,518 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:51,517 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,518 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:51,517 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:51,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:51,518 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,518 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:51,618 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:51,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:51,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:51,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:51,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:51,621 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:51,621 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:51,621 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:51,621 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:51,622 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,622 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,624 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,624 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:51,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,625 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,625 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:51,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:51,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:51,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:51,625 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:51,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:51,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:51,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:51,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:51,626 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:51,626 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:51,626 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:51,626 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:51,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:51,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:51,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:51,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:51,628 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,628 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:51,628 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,628 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:51,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:51,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e2c9d4c[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:51,629 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:51,629 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,630 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,630 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:51,631 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:51,631 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:51,632 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,632 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:51,633 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,633 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:51,633 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:51,633 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:51,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:51,635 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:51,635 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:51,635 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:51,635 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:51,635 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:51,641 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,641 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:51,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,641 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:51,641 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:51,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,643 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:51,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,645 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,645 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,645 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,645 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:51,645 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:51,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:51,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:51,645 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,645 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:51,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:51,646 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,646 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,646 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:51,646 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:51,646 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:51,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:51,746 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:51,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:51,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:51,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:51,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:51,749 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:51,749 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:51,749 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:51,749 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:51,750 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,750 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,751 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:51,751 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:51,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:51,752 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,752 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:51,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:51,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:51,753 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:51,753 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:51,753 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:51,753 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:51,753 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:51,753 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:51,753 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:51,753 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:51,753 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:51,753 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:51,753 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:51,753 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:51,753 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:51,753 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,754 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:51,754 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:51,755 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,755 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:51,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:51,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:51,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:51,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,756 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,756 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:51,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:51,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:51,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fd26991[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:51,757 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:51,757 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,760 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:51,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:51,760 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,760 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:51,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:51,761 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,762 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:51,762 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:51,762 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:51,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:51,764 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:51,764 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:51,764 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:51,764 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:51,764 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:51,766 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:51,766 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,766 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:51,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:51,767 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,768 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,771 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:51,771 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:51,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:51,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:51,771 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,771 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,771 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,771 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,771 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,771 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:51,771 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,772 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:51,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:51,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:51,772 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:51,772 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:51,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:51,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45982 deadline: 1682567811782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:51,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45952 deadline: 1682567811786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,851 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/28b838fcd47148d38a414739692b645e 2023-04-27 03:55:51,855 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/4876430557d04dec933a303f8b283248 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/4876430557d04dec933a303f8b283248 2023-04-27 03:55:51,858 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/4876430557d04dec933a303f8b283248, entries=200, sequenceid=443, filesize=14.3 K 2023-04-27 03:55:51,858 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/058cea19791749e6b7a1978d05ac0f2d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/058cea19791749e6b7a1978d05ac0f2d 2023-04-27 03:55:51,861 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/058cea19791749e6b7a1978d05ac0f2d, entries=150, sequenceid=443, filesize=11.9 K 2023-04-27 03:55:51,862 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/28b838fcd47148d38a414739692b645e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/28b838fcd47148d38a414739692b645e 2023-04-27 03:55:51,865 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/28b838fcd47148d38a414739692b645e, entries=150, sequenceid=443, filesize=11.9 K 2023-04-27 03:55:51,866 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 98cbc3f1a78abce9e99d0349272cd6a4 in 1244ms, sequenceid=443, compaction requested=true 2023-04-27 03:55:51,866 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:51,866 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:51,866 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:51,866 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:51,866 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:51,866 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:51,867 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39782 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:51,867 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/A is initiating minor compaction (all files) 2023-04-27 03:55:51,867 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:51,867 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/A in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:51,867 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/B is initiating minor compaction (all files) 2023-04-27 03:55:51,867 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/4261f2d032d3455db501d51c3d7deeff, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/2b8f6d4d445542bd80a39ba5ebf732b1, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/4876430557d04dec933a303f8b283248] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=38.8 K 2023-04-27 03:55:51,867 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/B in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:51,868 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/b2fd8fa7114c4078ac9dcfbf2aa48f1a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/c30501fbfdb24fdf90f05b35d070c82d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/058cea19791749e6b7a1978d05ac0f2d] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=36.5 K 2023-04-27 03:55:51,868 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 4261f2d032d3455db501d51c3d7deeff, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1682567749018 2023-04-27 03:55:51,868 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting b2fd8fa7114c4078ac9dcfbf2aa48f1a, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1682567749018 2023-04-27 03:55:51,868 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 2b8f6d4d445542bd80a39ba5ebf732b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1682567749330 2023-04-27 03:55:51,868 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting c30501fbfdb24fdf90f05b35d070c82d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1682567749330 2023-04-27 03:55:51,869 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 4876430557d04dec933a303f8b283248, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1682567749999 2023-04-27 03:55:51,869 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 058cea19791749e6b7a1978d05ac0f2d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1682567749999 2023-04-27 03:55:51,872 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:51,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:51,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:51,879 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#B#compaction#494 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:51,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:51,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:51,879 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:51,880 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:51,880 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:51,880 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:51,880 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,880 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,882 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#A#compaction#495 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:51,883 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:51,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:51,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:51,883 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,883 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:51,883 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:51,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:51,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:51,884 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:51,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:51,884 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:51,884 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:51,884 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:51,888 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:51,888 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:51,888 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:51,888 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:51,888 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-27 03:55:51,888 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:51,888 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:51,888 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:51,888 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:51,889 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:51,889 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:51,911 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/487d1ccd51404fa4b7653c35bcf92058 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/487d1ccd51404fa4b7653c35bcf92058 2023-04-27 03:55:51,915 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/b77894d64ed444e88784ae60af9375ed as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/b77894d64ed444e88784ae60af9375ed 2023-04-27 03:55:51,919 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/A of 98cbc3f1a78abce9e99d0349272cd6a4 into 487d1ccd51404fa4b7653c35bcf92058(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:51,919 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:51,919 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/A, priority=13, startTime=1682567751866; duration=0sec 2023-04-27 03:55:51,919 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:51,919 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:51,920 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/B of 98cbc3f1a78abce9e99d0349272cd6a4 into b77894d64ed444e88784ae60af9375ed(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:51,920 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:51,920 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/B, priority=13, startTime=1682567751866; duration=0sec 2023-04-27 03:55:51,921 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:51,921 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:51,921 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/C is initiating minor compaction (all files) 2023-04-27 03:55:51,921 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/C in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:51,921 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/100e85c16d67411d8fe30233be9c5931, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/3323f6a7c03c4d808dfd97ef2a4ebff2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/28b838fcd47148d38a414739692b645e] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=36.5 K 2023-04-27 03:55:51,922 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 100e85c16d67411d8fe30233be9c5931, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1682567749018 2023-04-27 03:55:51,922 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 3323f6a7c03c4d808dfd97ef2a4ebff2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1682567749330 2023-04-27 03:55:51,923 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 28b838fcd47148d38a414739692b645e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1682567749999 2023-04-27 03:55:51,929 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#C#compaction#497 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:51,936 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/b60f0a3743ae4d689c73f45d5b6d559e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/b60f0a3743ae4d689c73f45d5b6d559e 2023-04-27 03:55:51,940 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/C of 98cbc3f1a78abce9e99d0349272cd6a4 into b60f0a3743ae4d689c73f45d5b6d559e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:51,941 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:51,941 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/C, priority=13, startTime=1682567751866; duration=0sec 2023-04-27 03:55:51,941 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:52,325 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=457 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/4915496376424140980e22acfdd6baa0 2023-04-27 03:55:52,738 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=457 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/6d4031284b06463c92ccd39c8489496e 2023-04-27 03:55:52,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:52,792 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:52,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:52,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45982 deadline: 1682567812828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:52,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:52,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45952 deadline: 1682567812832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:52,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:52,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45982 deadline: 1682567812934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:52,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:52,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45952 deadline: 1682567812937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:53,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45982 deadline: 1682567813138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:53,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45952 deadline: 1682567813141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,148 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=457 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/e97158d2b8ce4b1ea4b05d396a64a1a2 2023-04-27 03:55:53,152 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/4915496376424140980e22acfdd6baa0 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/4915496376424140980e22acfdd6baa0 2023-04-27 03:55:53,155 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/4915496376424140980e22acfdd6baa0, entries=150, sequenceid=457, filesize=11.9 K 2023-04-27 03:55:53,156 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/6d4031284b06463c92ccd39c8489496e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/6d4031284b06463c92ccd39c8489496e 2023-04-27 03:55:53,159 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/6d4031284b06463c92ccd39c8489496e, entries=150, sequenceid=457, filesize=11.9 K 2023-04-27 03:55:53,160 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/e97158d2b8ce4b1ea4b05d396a64a1a2 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/e97158d2b8ce4b1ea4b05d396a64a1a2 2023-04-27 03:55:53,164 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/e97158d2b8ce4b1ea4b05d396a64a1a2, entries=150, sequenceid=457, filesize=11.9 K 2023-04-27 03:55:53,165 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 98cbc3f1a78abce9e99d0349272cd6a4 in 1277ms, sequenceid=457, compaction requested=false 2023-04-27 03:55:53,165 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:53,165 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:53,165 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:53,165 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:53,165 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,165 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:53,165 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:53,167 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,167 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:53,167 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,167 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:53,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:53,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c9f84a4[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:53,169 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:53,169 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,170 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,170 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:53,171 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:53,171 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,171 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:53,171 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:53,172 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:53,172 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:53,172 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,172 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:53,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:53,174 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:53,175 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:53,175 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:53,175 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:53,175 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:53,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:53,188 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-27 03:55:53,188 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:53,188 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:53,188 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:53,188 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,188 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:53,188 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:53,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,188 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:53,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:53,189 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:53,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:53,189 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:53,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,190 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:53,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:53,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:53,193 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,193 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,193 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,193 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,193 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,194 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,194 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,194 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:53,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:53,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:53,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:53,196 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:53,196 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:53,196 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:53,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:53,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:53,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:53,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45930 deadline: 1682567813209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:53,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45914 deadline: 1682567813210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:53,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567813211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,296 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:53,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:53,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:53,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:53,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:53,299 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:53,300 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:53,300 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:53,300 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:53,300 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,300 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,301 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:53,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:53,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:53,301 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,302 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,302 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:53,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:53,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:53,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:53,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:53,303 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:53,303 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:53,303 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:53,303 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:53,303 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:53,303 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:53,303 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:53,303 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:53,303 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:53,303 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:53,303 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:53,303 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,303 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:53,304 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:53,306 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,306 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:53,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:53,307 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,307 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:53,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:53,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:53,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:53,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:53,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bb9eaeb[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:53,308 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:53,308 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,309 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,309 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:53,309 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:53,309 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:53,310 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,310 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:53,311 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:53,311 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:53,311 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,311 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:53,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:53,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:53,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:53,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:53,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:53,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:53,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:53,313 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:53,313 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:53,313 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:53,313 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:53,314 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:53,315 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,315 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:53,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:53,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,315 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:53,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:53,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:53,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:53,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,316 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:53,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,317 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:53,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:53,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45930 deadline: 1682567813313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:53,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45914 deadline: 1682567813315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:53,320 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:53,320 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:53,320 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567813317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,320 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,320 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,320 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,320 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,320 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,320 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:53,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:53,320 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:53,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:53,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:53,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:53,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:53,321 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:53,321 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:53,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:53,421 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:53,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:53,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:53,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:53,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:53,424 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:53,424 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:53,424 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:53,424 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:53,425 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,425 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,426 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:53,426 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:53,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:53,426 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,426 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:53,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:53,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:53,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:53,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:53,427 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:53,427 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:53,427 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:53,427 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:53,428 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:53,428 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:53,428 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:53,428 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:53,428 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:53,428 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:53,428 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:53,428 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,428 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:53,428 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:53,429 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,429 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:53,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:53,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:53,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,430 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,430 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:53,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:53,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:53,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11c5c42d[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:53,431 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:53,431 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,432 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,432 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:53,432 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:53,432 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:53,432 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,433 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:53,435 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:53,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:53,435 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:53,435 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:53,435 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:53,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:53,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:53,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:53,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:53,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:53,437 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:53,438 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:53,438 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:53,438 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:53,438 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:53,439 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,439 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:53,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,439 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:53,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:53,440 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:53,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,441 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,444 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,444 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,444 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,444 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:53,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:53,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:53,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:53,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:53,445 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:53,445 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,445 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:53,445 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:53,445 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:53,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:53,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:53,445 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,445 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,445 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:53,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45982 deadline: 1682567813441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:53,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45952 deadline: 1682567813446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:53,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45930 deadline: 1682567813520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:53,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45914 deadline: 1682567813522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:53,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567813522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,545 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:53,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:53,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:53,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:53,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:53,548 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:53,548 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:53,548 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:53,549 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:53,549 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,549 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,550 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:53,550 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:53,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:53,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,551 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,551 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:53,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:53,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:53,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:53,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:53,552 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:53,552 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:53,552 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:53,553 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:53,553 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:53,553 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:53,553 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:53,553 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:53,553 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:53,553 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:53,553 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:53,553 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,553 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:53,553 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:53,556 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,556 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:53,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:53,557 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,557 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:53,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:53,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:53,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:53,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:53,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d57b965[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:53,558 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:53,558 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,559 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,559 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:53,559 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:53,559 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:53,560 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,560 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:53,561 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,561 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:53,561 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:53,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:53,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:53,561 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:53,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:53,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:53,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:53,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:53,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:53,563 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:53,563 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:53,563 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:53,563 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:53,563 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:53,565 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,565 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:53,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,565 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:53,566 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,567 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,572 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,572 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,572 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:53,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:53,572 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:53,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:53,573 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,572 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:53,573 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,573 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:53,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:53,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:53,573 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:53,573 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:53,573 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,573 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:53,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:53,602 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/6e998ecb1600478cb216d90d88066fa4 2023-04-27 03:55:53,612 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/00ebc35d493149609f097a92eeba4ec3 2023-04-27 03:55:53,673 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:53,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:53,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:53,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:53,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:53,677 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:53,677 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:53,677 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:53,677 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:53,677 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,677 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,678 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,678 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:53,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:53,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:53,679 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,679 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:53,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:53,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:53,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:53,681 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:53,681 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:53,681 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:53,682 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:53,682 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:53,682 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:53,682 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:53,682 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:53,682 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:53,682 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:53,682 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:53,682 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,682 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:53,682 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:53,683 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,683 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:53,684 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,684 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:53,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:53,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dfda4cf[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:53,685 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:53,685 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,686 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,686 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:53,686 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,686 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:53,686 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:53,686 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:53,693 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:53,693 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:53,693 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,693 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:53,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:53,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:53,696 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:53,696 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:53,696 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:53,696 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:53,696 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:53,698 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,698 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,698 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:53,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:53,699 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,700 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,702 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,702 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:53,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:53,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:53,702 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,702 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:53,702 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:53,703 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:53,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:53,703 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,703 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:53,703 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,703 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:53,703 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:53,703 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:53,803 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:53,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:53,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:53,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:53,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:53,807 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:53,807 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:53,807 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:53,807 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:53,812 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,812 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,814 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:53,814 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,815 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,815 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:53,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:53,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:53,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:53,816 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:53,816 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:53,816 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:53,817 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:53,817 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:53,817 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:53,817 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:53,820 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,820 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:53,821 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,821 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:53,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:53,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6977f929[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:53,822 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:53,822 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,824 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,825 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:53,825 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:53,825 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:53,825 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,825 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:53,826 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:53,826 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,826 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:53,826 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:53,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:53,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:53,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45914 deadline: 1682567813824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:53,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,828 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:53,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567813825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:53,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45930 deadline: 1682567813826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:53,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:53,829 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:53,829 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:53,829 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:53,829 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:53,829 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:53,831 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,831 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,831 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:53,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:53,832 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,833 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,836 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,836 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,836 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,836 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,836 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,836 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,836 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,836 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:53,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:53,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:53,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:53,836 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:53,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:53,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:53,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:53,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:53,836 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:53,838 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:53,838 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:53,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:53,938 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:53,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:53,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:53,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:53,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:53,941 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:53,941 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:53,941 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:53,941 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:53,942 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,942 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,943 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:53,943 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:53,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:53,943 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,943 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:53,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:53,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:55:53,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:53,944 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:53,944 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:53,944 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:53,944 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:53,945 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:53,946 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:53,946 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:53,946 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:53,946 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:53,948 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,948 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:53,948 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,948 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:53,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:53,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56acbc86[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:53,950 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:53,950 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,951 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,951 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:53,951 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:53,951 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:53,951 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,951 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:53,954 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,954 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:53,954 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:53,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:53,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:53,954 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:53,956 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45982 deadline: 1682567813950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:53,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:53,957 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:53,957 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:53,957 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:53,957 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:53,957 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:53,959 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,959 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:53,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,959 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:53,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:53,960 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,961 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:53,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45952 deadline: 1682567813955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,966 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:53,966 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:53,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:53,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:53,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:53,967 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:53,967 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,967 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:53,967 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:53,966 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:53,967 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:53,967 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:53,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:53,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:53,967 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,967 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:53,967 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:54,023 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/7cc58619c9aa4a4dba0e764dba799aa9 2023-04-27 03:55:54,027 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/6e998ecb1600478cb216d90d88066fa4 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/6e998ecb1600478cb216d90d88066fa4 2023-04-27 03:55:54,031 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/6e998ecb1600478cb216d90d88066fa4, entries=250, sequenceid=480, filesize=16.7 K 2023-04-27 03:55:54,032 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/00ebc35d493149609f097a92eeba4ec3 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/00ebc35d493149609f097a92eeba4ec3 2023-04-27 03:55:54,035 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/00ebc35d493149609f097a92eeba4ec3, entries=150, sequenceid=480, filesize=11.9 K 2023-04-27 03:55:54,035 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/7cc58619c9aa4a4dba0e764dba799aa9 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/7cc58619c9aa4a4dba0e764dba799aa9 2023-04-27 03:55:54,038 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/7cc58619c9aa4a4dba0e764dba799aa9, entries=150, sequenceid=480, filesize=11.9 K 2023-04-27 03:55:54,039 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 98cbc3f1a78abce9e99d0349272cd6a4 in 851ms, sequenceid=480, compaction requested=true 2023-04-27 03:55:54,039 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:54,039 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:54,039 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:54,039 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:54,039 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:54,039 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:54,040 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42324 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:54,040 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:54,040 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/A is initiating minor compaction (all files) 2023-04-27 03:55:54,040 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/B is initiating minor compaction (all files) 2023-04-27 03:55:54,040 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/A in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:54,040 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/B in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:54,040 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/487d1ccd51404fa4b7653c35bcf92058, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/4915496376424140980e22acfdd6baa0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/6e998ecb1600478cb216d90d88066fa4] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=41.3 K 2023-04-27 03:55:54,040 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/b77894d64ed444e88784ae60af9375ed, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/6d4031284b06463c92ccd39c8489496e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/00ebc35d493149609f097a92eeba4ec3] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=36.6 K 2023-04-27 03:55:54,041 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 487d1ccd51404fa4b7653c35bcf92058, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1682567749999 2023-04-27 03:55:54,041 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting b77894d64ed444e88784ae60af9375ed, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1682567749999 2023-04-27 03:55:54,041 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 4915496376424140980e22acfdd6baa0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=457, earliestPutTs=1682567750658 2023-04-27 03:55:54,041 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 6d4031284b06463c92ccd39c8489496e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=457, earliestPutTs=1682567750658 2023-04-27 03:55:54,041 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 6e998ecb1600478cb216d90d88066fa4, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1682567752808 2023-04-27 03:55:54,041 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 00ebc35d493149609f097a92eeba4ec3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1682567752827 2023-04-27 03:55:54,047 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#B#compaction#504 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:54,047 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#A#compaction#503 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:54,067 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:54,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:54,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:55:54,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:55:54,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:55:54,070 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:55:54,070 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:54,070 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:55:54,071 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:55:54,071 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:54,071 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:54,074 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:54,074 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:54,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:54,074 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:54,074 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:55:54,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:55:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:54,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:55:54,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:55:54,075 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:55:54,075 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:55:54,075 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:55:54,076 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:55:54,076 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:55:54,076 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:54,076 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. started... 2023-04-27 03:55:54,077 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-27 03:55:54,077 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:54,077 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:54,077 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:54,077 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:54,077 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:54,077 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:54,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:54,335 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. as already flushing 2023-04-27 03:55:54,373 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:54,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45930 deadline: 1682567814364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:54,378 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:54,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45914 deadline: 1682567814372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:54,378 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:54,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567814373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:54,473 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/c36cce9ef173436eb37e4a30e741cb92 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/c36cce9ef173436eb37e4a30e741cb92 2023-04-27 03:55:54,474 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/b95ee5ba6ebf49f492b0b6abdc024b43 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/b95ee5ba6ebf49f492b0b6abdc024b43 2023-04-27 03:55:54,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:54,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45930 deadline: 1682567814474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:54,479 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/B of 98cbc3f1a78abce9e99d0349272cd6a4 into c36cce9ef173436eb37e4a30e741cb92(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:54,479 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:54,479 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/B, priority=13, startTime=1682567754039; duration=0sec 2023-04-27 03:55:54,479 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:55:54,479 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:55:54,480 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/A of 98cbc3f1a78abce9e99d0349272cd6a4 into b95ee5ba6ebf49f492b0b6abdc024b43(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:54,480 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:54,480 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/A, priority=13, startTime=1682567754039; duration=0sec 2023-04-27 03:55:54,480 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:54,480 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:55:54,481 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 98cbc3f1a78abce9e99d0349272cd6a4/C is initiating minor compaction (all files) 2023-04-27 03:55:54,481 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 98cbc3f1a78abce9e99d0349272cd6a4/C in TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:54,481 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/b60f0a3743ae4d689c73f45d5b6d559e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/e97158d2b8ce4b1ea4b05d396a64a1a2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/7cc58619c9aa4a4dba0e764dba799aa9] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp, totalSize=36.6 K 2023-04-27 03:55:54,481 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting b60f0a3743ae4d689c73f45d5b6d559e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1682567749999 2023-04-27 03:55:54,481 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting e97158d2b8ce4b1ea4b05d396a64a1a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=457, earliestPutTs=1682567750658 2023-04-27 03:55:54,482 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 7cc58619c9aa4a4dba0e764dba799aa9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1682567752827 2023-04-27 03:55:54,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:54,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45914 deadline: 1682567814479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:54,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:55:54,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45932 deadline: 1682567814480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:54,486 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 98cbc3f1a78abce9e99d0349272cd6a4#C#compaction#506 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:55:54,491 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/46ddd1f56279400d8785c18b33a8e975 2023-04-27 03:55:54,496 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/db4a7728d4af446ab1aa2ad85961b2bd as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/db4a7728d4af446ab1aa2ad85961b2bd 2023-04-27 03:55:54,505 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 98cbc3f1a78abce9e99d0349272cd6a4/C of 98cbc3f1a78abce9e99d0349272cd6a4 into db4a7728d4af446ab1aa2ad85961b2bd(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:55:54,505 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:54,505 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4., storeName=98cbc3f1a78abce9e99d0349272cd6a4/C, priority=13, startTime=1682567754039; duration=0sec 2023-04-27 03:55:54,505 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:54,506 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/749c8208a61d4064a557ad5e7bb4599d 2023-04-27 03:55:54,513 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/56f103b7bb114c22bacc4c56c0acc451 2023-04-27 03:55:54,516 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/46ddd1f56279400d8785c18b33a8e975 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/46ddd1f56279400d8785c18b33a8e975 2023-04-27 03:55:54,519 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/46ddd1f56279400d8785c18b33a8e975, entries=100, sequenceid=492, filesize=9.5 K 2023-04-27 03:55:54,519 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/749c8208a61d4064a557ad5e7bb4599d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/749c8208a61d4064a557ad5e7bb4599d 2023-04-27 03:55:54,522 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/749c8208a61d4064a557ad5e7bb4599d, entries=100, sequenceid=492, filesize=9.5 K 2023-04-27 03:55:54,522 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/56f103b7bb114c22bacc4c56c0acc451 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/56f103b7bb114c22bacc4c56c0acc451 2023-04-27 03:55:54,528 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/56f103b7bb114c22bacc4c56c0acc451, entries=100, sequenceid=492, filesize=9.5 K 2023-04-27 03:55:54,529 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 98cbc3f1a78abce9e99d0349272cd6a4 in 452ms, sequenceid=492, compaction requested=false 2023-04-27 03:55:54,529 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:54,529 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:54,529 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:55:54,529 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:55:54,529 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:54,529 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:55:54,529 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:55:54,531 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:54,531 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:54,531 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:54,531 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:55:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:54,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:55:54,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7690bba7[Count = 0] remaining members to acquire global barrier 2023-04-27 03:55:54,533 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:55:54,533 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:54,534 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:54,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:54,535 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:55:54,535 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:55:54,535 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:54,535 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:55:54,535 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:55:54,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:54,538 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:55:54,538 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:54,538 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:55:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:54,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:54,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:55:54,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:55:54,539 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:55:54,539 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:55:54,539 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:55:54,539 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:55:54,539 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:55:54,542 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:54,542 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:54,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:54,542 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:55:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:55:54,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:54,542 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:55:54,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:54,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:54,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:55:54,544 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:54,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:55:54,546 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:54,546 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:54,546 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:55:54,546 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:55:54,546 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:54,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:55:54,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:55:54,546 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:55:54,546 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:55:54,547 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:55:54,547 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:55:54,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:55:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:55:54,547 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:55:54,547 DEBUG [Thread-3258] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:55:54,547 DEBUG [Thread-3258] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:55:54,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:55:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:55:54,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:55:54,609 DEBUG [Thread-3263] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x659c68ab to 127.0.0.1:55020 2023-04-27 03:55:54,609 DEBUG [Thread-3263] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:55:54,609 DEBUG [Thread-3261] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78592df0 to 127.0.0.1:55020 2023-04-27 03:55:54,609 DEBUG [Thread-3261] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:55:54,611 DEBUG [Thread-3267] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b8f652e to 127.0.0.1:55020 2023-04-27 03:55:54,611 DEBUG [Thread-3267] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:55:54,612 DEBUG [Thread-3265] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x680e773e to 127.0.0.1:55020 2023-04-27 03:55:54,612 DEBUG [Thread-3265] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:55:54,613 DEBUG [Thread-3259] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x067ea274 to 127.0.0.1:55020 2023-04-27 03:55:54,613 DEBUG [Thread-3259] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:55:54,647 DEBUG [Thread-3258] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:55:54,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:55:54,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:54,680 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-27 03:55:54,680 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:54,680 DEBUG [Thread-3256] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x67346c47 to 127.0.0.1:55020 2023-04-27 03:55:54,681 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:54,681 DEBUG [Thread-3256] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:55:54,681 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:54,681 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:54,681 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:54,681 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:55:54,688 DEBUG [Thread-3248] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15880ffb to 127.0.0.1:55020 2023-04-27 03:55:54,688 DEBUG [Thread-3250] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x052203cb to 127.0.0.1:55020 2023-04-27 03:55:54,688 DEBUG [Thread-3248] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:55:54,688 DEBUG [Thread-3250] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:55:54,962 DEBUG [Thread-3252] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x420f945a to 127.0.0.1:55020 2023-04-27 03:55:54,962 DEBUG [Thread-3252] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:55:54,973 DEBUG [Thread-3254] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51b8df2a to 127.0.0.1:55020 2023-04-27 03:55:54,973 DEBUG [Thread-3254] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:55:54,973 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-27 03:55:54,973 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 109 2023-04-27 03:55:54,973 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 90 2023-04-27 03:55:54,973 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 63 2023-04-27 03:55:54,973 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 57 2023-04-27 03:55:54,973 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 83 2023-04-27 03:55:54,973 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-27 03:55:54,973 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-27 03:55:54,973 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(402): scanned 1888 2023-04-27 03:55:54,973 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(403): verified 5661 rows 2023-04-27 03:55:54,973 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(402): scanned 1885 2023-04-27 03:55:54,973 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(403): verified 5652 rows 2023-04-27 03:55:54,973 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(402): scanned 1891 2023-04-27 03:55:54,973 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(403): verified 5673 rows 2023-04-27 03:55:54,974 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(402): scanned 1894 2023-04-27 03:55:54,974 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(403): verified 5679 rows 2023-04-27 03:55:54,974 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(402): scanned 1907 2023-04-27 03:55:54,974 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(403): verified 5718 rows 2023-04-27 03:55:54,974 INFO [Listener at localhost/33933] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-27 03:55:54,974 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x37705401 to 127.0.0.1:55020 2023-04-27 03:55:54,974 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:55:54,975 INFO [Listener at localhost/33933] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-27 03:55:54,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-27 03:55:54,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-27 03:55:54,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-27 03:55:54,982 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567754982"}]},"ts":"1682567754982"} 2023-04-27 03:55:54,983 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-27 03:55:54,985 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-27 03:55:54,985 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=98cbc3f1a78abce9e99d0349272cd6a4, UNASSIGN}] 2023-04-27 03:55:54,987 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=98cbc3f1a78abce9e99d0349272cd6a4, UNASSIGN 2023-04-27 03:55:54,987 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=98cbc3f1a78abce9e99d0349272cd6a4, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:55:54,987 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567754987"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682567754987"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682567754987"}]},"ts":"1682567754987"} 2023-04-27 03:55:54,988 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305}] 2023-04-27 03:55:55,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-27 03:55:55,087 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/7cd32992135e4674b11e3932c4a7f070 2023-04-27 03:55:55,140 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:55,140 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 98cbc3f1a78abce9e99d0349272cd6a4, disabling compactions & flushes 2023-04-27 03:55:55,140 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:55,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-27 03:55:55,495 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/8835cdad58394ea7aff9c5bdc82f1348 2023-04-27 03:55:55,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-27 03:55:55,904 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/fa1e4305785c4250abee9c9682156057 2023-04-27 03:55:55,907 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/7cd32992135e4674b11e3932c4a7f070 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/7cd32992135e4674b11e3932c4a7f070 2023-04-27 03:55:55,910 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/7cd32992135e4674b11e3932c4a7f070, entries=150, sequenceid=516, filesize=11.9 K 2023-04-27 03:55:55,910 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/8835cdad58394ea7aff9c5bdc82f1348 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/8835cdad58394ea7aff9c5bdc82f1348 2023-04-27 03:55:55,913 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/8835cdad58394ea7aff9c5bdc82f1348, entries=150, sequenceid=516, filesize=11.9 K 2023-04-27 03:55:55,914 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/fa1e4305785c4250abee9c9682156057 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/fa1e4305785c4250abee9c9682156057 2023-04-27 03:55:55,916 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/fa1e4305785c4250abee9c9682156057, entries=150, sequenceid=516, filesize=11.9 K 2023-04-27 03:55:55,917 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=26.84 KB/27480 for 98cbc3f1a78abce9e99d0349272cd6a4 in 1237ms, sequenceid=516, compaction requested=true 2023-04-27 03:55:55,917 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:55,917 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:55,917 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. because compaction request was cancelled 2023-04-27 03:55:55,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:55,917 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:55,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. after waiting 0 ms 2023-04-27 03:55:55,917 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. because compaction request was cancelled 2023-04-27 03:55:55,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:55,917 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:55,917 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:55:55,917 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 98cbc3f1a78abce9e99d0349272cd6a4 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-27 03:55:55,917 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. because compaction request was cancelled 2023-04-27 03:55:55,918 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=A 2023-04-27 03:55:55,918 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:55,918 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=B 2023-04-27 03:55:55,918 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:55,918 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 98cbc3f1a78abce9e99d0349272cd6a4, store=C 2023-04-27 03:55:55,918 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:55:56,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-27 03:55:56,325 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/fc364bba45b54c49a9773e96075e4f9a 2023-04-27 03:55:56,734 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/ca88aeaa8cbb4ca3baf8d1c1824700f8 2023-04-27 03:55:57,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-27 03:55:57,144 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/0777ed50cfbb45a4829f6a9ad0c0479a 2023-04-27 03:55:57,147 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/A/fc364bba45b54c49a9773e96075e4f9a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/fc364bba45b54c49a9773e96075e4f9a 2023-04-27 03:55:57,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/fc364bba45b54c49a9773e96075e4f9a, entries=100, sequenceid=523, filesize=9.5 K 2023-04-27 03:55:57,151 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/B/ca88aeaa8cbb4ca3baf8d1c1824700f8 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/ca88aeaa8cbb4ca3baf8d1c1824700f8 2023-04-27 03:55:57,153 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/ca88aeaa8cbb4ca3baf8d1c1824700f8, entries=100, sequenceid=523, filesize=9.5 K 2023-04-27 03:55:57,154 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/.tmp/C/0777ed50cfbb45a4829f6a9ad0c0479a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/0777ed50cfbb45a4829f6a9ad0c0479a 2023-04-27 03:55:57,157 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/0777ed50cfbb45a4829f6a9ad0c0479a, entries=100, sequenceid=523, filesize=9.5 K 2023-04-27 03:55:57,158 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 98cbc3f1a78abce9e99d0349272cd6a4 in 1241ms, sequenceid=523, compaction requested=true 2023-04-27 03:55:57,183 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/b798b1fda6cb4ab68f09c6f650d337ac, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/be3bb316bfed451f9839df444860fa6c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/97ef251156094006bf37bf156e1825b4, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/a63962a56ca74007b15dddfa151d74c9, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/38eb9f2769e14718900f2b56c6abea59, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/2f66038a23574b7f86fba7d5e7409021, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/a14b8d58f9004774ac184eff7d961410, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/d3c3e9a063c94dc5a77e91c5f19efb11, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/b8daaed48fb94e6c8f60731cc39d6083, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/a4bf7b54ee15466aa49c91d0e9f0172f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/345246d6362b45dea1eb62a9b6ee48c3, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/74a1eac81ab645b1b814c567c790833c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/8b03cffb0ac9490b80f417d63a3b98d6, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/5074919511504133a6f3abcf7a731f4f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/032bd2fbeea2434f831bc3d7b65f9a30, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/94701c92c8974cc2be338eab122938c0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/71cfff43b2854413ae5f504cdf5154cf, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/666ec23544a14fa08ad1af5b4bed0fed, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/7f04439000c24e7ca2b276db45f8a294, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/936a3440000c44cebc29794c077a764d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/bee61f531fb0439aa9ff7a6f14b443b5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/516f58a0dd554c848a2831fa4c638838, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/0cb7409bfd344f2a932629ebdc135069, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/4261f2d032d3455db501d51c3d7deeff, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/2b8f6d4d445542bd80a39ba5ebf732b1, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/4876430557d04dec933a303f8b283248, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/487d1ccd51404fa4b7653c35bcf92058, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/4915496376424140980e22acfdd6baa0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/6e998ecb1600478cb216d90d88066fa4] to archive 2023-04-27 03:55:57,184 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 03:55:57,185 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/b798b1fda6cb4ab68f09c6f650d337ac to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/b798b1fda6cb4ab68f09c6f650d337ac 2023-04-27 03:55:57,186 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/be3bb316bfed451f9839df444860fa6c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/be3bb316bfed451f9839df444860fa6c 2023-04-27 03:55:57,187 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/97ef251156094006bf37bf156e1825b4 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/97ef251156094006bf37bf156e1825b4 2023-04-27 03:55:57,188 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/a63962a56ca74007b15dddfa151d74c9 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/a63962a56ca74007b15dddfa151d74c9 2023-04-27 03:55:57,188 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/38eb9f2769e14718900f2b56c6abea59 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/38eb9f2769e14718900f2b56c6abea59 2023-04-27 03:55:57,189 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/2f66038a23574b7f86fba7d5e7409021 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/2f66038a23574b7f86fba7d5e7409021 2023-04-27 03:55:57,190 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/a14b8d58f9004774ac184eff7d961410 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/a14b8d58f9004774ac184eff7d961410 2023-04-27 03:55:57,191 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/d3c3e9a063c94dc5a77e91c5f19efb11 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/d3c3e9a063c94dc5a77e91c5f19efb11 2023-04-27 03:55:57,191 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/b8daaed48fb94e6c8f60731cc39d6083 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/b8daaed48fb94e6c8f60731cc39d6083 2023-04-27 03:55:57,192 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/a4bf7b54ee15466aa49c91d0e9f0172f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/a4bf7b54ee15466aa49c91d0e9f0172f 2023-04-27 03:55:57,193 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/345246d6362b45dea1eb62a9b6ee48c3 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/345246d6362b45dea1eb62a9b6ee48c3 2023-04-27 03:55:57,194 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/74a1eac81ab645b1b814c567c790833c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/74a1eac81ab645b1b814c567c790833c 2023-04-27 03:55:57,195 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/8b03cffb0ac9490b80f417d63a3b98d6 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/8b03cffb0ac9490b80f417d63a3b98d6 2023-04-27 03:55:57,195 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/5074919511504133a6f3abcf7a731f4f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/5074919511504133a6f3abcf7a731f4f 2023-04-27 03:55:57,196 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/032bd2fbeea2434f831bc3d7b65f9a30 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/032bd2fbeea2434f831bc3d7b65f9a30 2023-04-27 03:55:57,197 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/94701c92c8974cc2be338eab122938c0 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/94701c92c8974cc2be338eab122938c0 2023-04-27 03:55:57,198 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/71cfff43b2854413ae5f504cdf5154cf to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/71cfff43b2854413ae5f504cdf5154cf 2023-04-27 03:55:57,199 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/666ec23544a14fa08ad1af5b4bed0fed to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/666ec23544a14fa08ad1af5b4bed0fed 2023-04-27 03:55:57,200 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/7f04439000c24e7ca2b276db45f8a294 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/7f04439000c24e7ca2b276db45f8a294 2023-04-27 03:55:57,201 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/936a3440000c44cebc29794c077a764d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/936a3440000c44cebc29794c077a764d 2023-04-27 03:55:57,201 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/bee61f531fb0439aa9ff7a6f14b443b5 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/bee61f531fb0439aa9ff7a6f14b443b5 2023-04-27 03:55:57,202 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/516f58a0dd554c848a2831fa4c638838 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/516f58a0dd554c848a2831fa4c638838 2023-04-27 03:55:57,203 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/0cb7409bfd344f2a932629ebdc135069 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/0cb7409bfd344f2a932629ebdc135069 2023-04-27 03:55:57,203 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/4261f2d032d3455db501d51c3d7deeff to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/4261f2d032d3455db501d51c3d7deeff 2023-04-27 03:55:57,204 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/2b8f6d4d445542bd80a39ba5ebf732b1 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/2b8f6d4d445542bd80a39ba5ebf732b1 2023-04-27 03:55:57,205 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/4876430557d04dec933a303f8b283248 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/4876430557d04dec933a303f8b283248 2023-04-27 03:55:57,205 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/487d1ccd51404fa4b7653c35bcf92058 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/487d1ccd51404fa4b7653c35bcf92058 2023-04-27 03:55:57,206 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/4915496376424140980e22acfdd6baa0 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/4915496376424140980e22acfdd6baa0 2023-04-27 03:55:57,207 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/6e998ecb1600478cb216d90d88066fa4 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/6e998ecb1600478cb216d90d88066fa4 2023-04-27 03:55:57,209 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/77dbc1cc0fb649c3811c1488a1fd86d6, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/21a6756e47074d6dbb50430a1b1a8c7c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/d6b6b1f57c67492d860036d1f5325f94, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/86d15e64f2354e249a8622e41fe51fb9, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/19974a6638f34a6187fad2e600337ff5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/5e604011c20b4cdf970951646855ceb3, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/a453869a5d9a4172adff2fd35412eb58, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/3f2c5ff6759c4c8094668bb6b00c8136, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/8f83895db57648f0a931ad5aa505de26, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/1292fcab792f4876a3809a42313159d5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/dddb0b9c6d0645189cd85622b8c2b186, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/97c4ba2b20064700801b023518d64de7, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/cd0962cedc8c4f6a9bcbdc07f8233073, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/f24416ec3ede4c68bc924c928292587c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/a8723498be7b4d41bb1334b1d3f52d86, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/8140ffd2db4b4dab9c145a36583cba17, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/2f6644c85d254562bef60e7d7cc07de8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/9a5f1e88e81f400997176a8c5880f0b3, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/37281145b4e349f78df4435b83f70a71, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/c0cdd5700eca4a0e8841b3e32c85b5b5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/0ee4887afff847c0bb6cf32285316422, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/4cf3969ff9214f5b85ac6b7cbc82cd57, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/b2fd8fa7114c4078ac9dcfbf2aa48f1a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/7dc9ce56c32d42e686ecd4a347033503, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/c30501fbfdb24fdf90f05b35d070c82d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/b77894d64ed444e88784ae60af9375ed, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/058cea19791749e6b7a1978d05ac0f2d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/6d4031284b06463c92ccd39c8489496e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/00ebc35d493149609f097a92eeba4ec3] to archive 2023-04-27 03:55:57,210 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 03:55:57,211 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/77dbc1cc0fb649c3811c1488a1fd86d6 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/77dbc1cc0fb649c3811c1488a1fd86d6 2023-04-27 03:55:57,212 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/21a6756e47074d6dbb50430a1b1a8c7c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/21a6756e47074d6dbb50430a1b1a8c7c 2023-04-27 03:55:57,213 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/d6b6b1f57c67492d860036d1f5325f94 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/d6b6b1f57c67492d860036d1f5325f94 2023-04-27 03:55:57,214 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/86d15e64f2354e249a8622e41fe51fb9 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/86d15e64f2354e249a8622e41fe51fb9 2023-04-27 03:55:57,215 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/19974a6638f34a6187fad2e600337ff5 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/19974a6638f34a6187fad2e600337ff5 2023-04-27 03:55:57,216 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/5e604011c20b4cdf970951646855ceb3 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/5e604011c20b4cdf970951646855ceb3 2023-04-27 03:55:57,217 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/a453869a5d9a4172adff2fd35412eb58 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/a453869a5d9a4172adff2fd35412eb58 2023-04-27 03:55:57,218 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/3f2c5ff6759c4c8094668bb6b00c8136 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/3f2c5ff6759c4c8094668bb6b00c8136 2023-04-27 03:55:57,219 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/8f83895db57648f0a931ad5aa505de26 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/8f83895db57648f0a931ad5aa505de26 2023-04-27 03:55:57,220 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/1292fcab792f4876a3809a42313159d5 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/1292fcab792f4876a3809a42313159d5 2023-04-27 03:55:57,221 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/dddb0b9c6d0645189cd85622b8c2b186 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/dddb0b9c6d0645189cd85622b8c2b186 2023-04-27 03:55:57,222 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/97c4ba2b20064700801b023518d64de7 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/97c4ba2b20064700801b023518d64de7 2023-04-27 03:55:57,223 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/cd0962cedc8c4f6a9bcbdc07f8233073 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/cd0962cedc8c4f6a9bcbdc07f8233073 2023-04-27 03:55:57,224 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/f24416ec3ede4c68bc924c928292587c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/f24416ec3ede4c68bc924c928292587c 2023-04-27 03:55:57,225 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/a8723498be7b4d41bb1334b1d3f52d86 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/a8723498be7b4d41bb1334b1d3f52d86 2023-04-27 03:55:57,226 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/8140ffd2db4b4dab9c145a36583cba17 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/8140ffd2db4b4dab9c145a36583cba17 2023-04-27 03:55:57,227 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/2f6644c85d254562bef60e7d7cc07de8 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/2f6644c85d254562bef60e7d7cc07de8 2023-04-27 03:55:57,228 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/9a5f1e88e81f400997176a8c5880f0b3 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/9a5f1e88e81f400997176a8c5880f0b3 2023-04-27 03:55:57,229 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/37281145b4e349f78df4435b83f70a71 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/37281145b4e349f78df4435b83f70a71 2023-04-27 03:55:57,230 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/c0cdd5700eca4a0e8841b3e32c85b5b5 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/c0cdd5700eca4a0e8841b3e32c85b5b5 2023-04-27 03:55:57,231 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/0ee4887afff847c0bb6cf32285316422 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/0ee4887afff847c0bb6cf32285316422 2023-04-27 03:55:57,232 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/4cf3969ff9214f5b85ac6b7cbc82cd57 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/4cf3969ff9214f5b85ac6b7cbc82cd57 2023-04-27 03:55:57,233 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/b2fd8fa7114c4078ac9dcfbf2aa48f1a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/b2fd8fa7114c4078ac9dcfbf2aa48f1a 2023-04-27 03:55:57,234 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/7dc9ce56c32d42e686ecd4a347033503 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/7dc9ce56c32d42e686ecd4a347033503 2023-04-27 03:55:57,235 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/c30501fbfdb24fdf90f05b35d070c82d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/c30501fbfdb24fdf90f05b35d070c82d 2023-04-27 03:55:57,236 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/b77894d64ed444e88784ae60af9375ed to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/b77894d64ed444e88784ae60af9375ed 2023-04-27 03:55:57,237 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/058cea19791749e6b7a1978d05ac0f2d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/058cea19791749e6b7a1978d05ac0f2d 2023-04-27 03:55:57,237 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/6d4031284b06463c92ccd39c8489496e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/6d4031284b06463c92ccd39c8489496e 2023-04-27 03:55:57,238 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/00ebc35d493149609f097a92eeba4ec3 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/00ebc35d493149609f097a92eeba4ec3 2023-04-27 03:55:57,241 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/dc21c6e1598a4308ad28470f893ebf9c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/c681f8685e7d455a9f61ddfb9451531e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/6eb198f1655e429a813e22a3852428ae, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/8e691d70025f40718b9d060095c96d91, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/8a6b439808544a3ebe2377ccdbd47ed2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/f09518bafff04883bc48eb6460045db3, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/365cb20a073049b2a87c1c3af1111c25, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/a1019f6cf2a44d97b598f0aeee6fcced, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/491847d70a67486e8339d9946ee3bbbd, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/a49da27384144a3297a36166a2c6f597, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/8f076a24ebe3479e9ba66c6adafc777c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/882d1ebb8fa049c58e1aa3b44f352f46, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/ab20fab8f0994609b05b06f45d965029, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/5e6f8b4eac924e788a1f814e6b3aa776, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/67d0415488f241eda61d42534aaf9980, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/0bb03e5c3a6a4f8089b6cef91148d0cc, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/1c769005cfc04fba93c476c6cc8f023e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/58fd4e32f33d4ceda8cf49d478b2acbf, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/d9a98cf7f3124ba2a31d02bcb07edd0e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/e5b2fabf84b843a59861a8463acf332d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/cbbda517f96e4bd684a943f381c1882c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/b9361a27edbe4cb18fac13cd54b197c5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/100e85c16d67411d8fe30233be9c5931, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/513df438a72649538dbfee7f48d8c5df, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/3323f6a7c03c4d808dfd97ef2a4ebff2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/b60f0a3743ae4d689c73f45d5b6d559e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/28b838fcd47148d38a414739692b645e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/e97158d2b8ce4b1ea4b05d396a64a1a2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/7cc58619c9aa4a4dba0e764dba799aa9] to archive 2023-04-27 03:55:57,242 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 03:55:57,243 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/dc21c6e1598a4308ad28470f893ebf9c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/dc21c6e1598a4308ad28470f893ebf9c 2023-04-27 03:55:57,244 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/c681f8685e7d455a9f61ddfb9451531e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/c681f8685e7d455a9f61ddfb9451531e 2023-04-27 03:55:57,245 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/6eb198f1655e429a813e22a3852428ae to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/6eb198f1655e429a813e22a3852428ae 2023-04-27 03:55:57,246 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/8e691d70025f40718b9d060095c96d91 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/8e691d70025f40718b9d060095c96d91 2023-04-27 03:55:57,246 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/8a6b439808544a3ebe2377ccdbd47ed2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/8a6b439808544a3ebe2377ccdbd47ed2 2023-04-27 03:55:57,247 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/f09518bafff04883bc48eb6460045db3 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/f09518bafff04883bc48eb6460045db3 2023-04-27 03:55:57,248 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/365cb20a073049b2a87c1c3af1111c25 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/365cb20a073049b2a87c1c3af1111c25 2023-04-27 03:55:57,249 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/a1019f6cf2a44d97b598f0aeee6fcced to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/a1019f6cf2a44d97b598f0aeee6fcced 2023-04-27 03:55:57,249 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/491847d70a67486e8339d9946ee3bbbd to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/491847d70a67486e8339d9946ee3bbbd 2023-04-27 03:55:57,250 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/a49da27384144a3297a36166a2c6f597 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/a49da27384144a3297a36166a2c6f597 2023-04-27 03:55:57,251 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/8f076a24ebe3479e9ba66c6adafc777c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/8f076a24ebe3479e9ba66c6adafc777c 2023-04-27 03:55:57,252 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/882d1ebb8fa049c58e1aa3b44f352f46 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/882d1ebb8fa049c58e1aa3b44f352f46 2023-04-27 03:55:57,252 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/ab20fab8f0994609b05b06f45d965029 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/ab20fab8f0994609b05b06f45d965029 2023-04-27 03:55:57,253 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/5e6f8b4eac924e788a1f814e6b3aa776 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/5e6f8b4eac924e788a1f814e6b3aa776 2023-04-27 03:55:57,254 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/67d0415488f241eda61d42534aaf9980 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/67d0415488f241eda61d42534aaf9980 2023-04-27 03:55:57,255 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/0bb03e5c3a6a4f8089b6cef91148d0cc to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/0bb03e5c3a6a4f8089b6cef91148d0cc 2023-04-27 03:55:57,255 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/1c769005cfc04fba93c476c6cc8f023e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/1c769005cfc04fba93c476c6cc8f023e 2023-04-27 03:55:57,256 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/58fd4e32f33d4ceda8cf49d478b2acbf to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/58fd4e32f33d4ceda8cf49d478b2acbf 2023-04-27 03:55:57,257 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/d9a98cf7f3124ba2a31d02bcb07edd0e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/d9a98cf7f3124ba2a31d02bcb07edd0e 2023-04-27 03:55:57,258 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/e5b2fabf84b843a59861a8463acf332d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/e5b2fabf84b843a59861a8463acf332d 2023-04-27 03:55:57,258 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/cbbda517f96e4bd684a943f381c1882c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/cbbda517f96e4bd684a943f381c1882c 2023-04-27 03:55:57,259 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/b9361a27edbe4cb18fac13cd54b197c5 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/b9361a27edbe4cb18fac13cd54b197c5 2023-04-27 03:55:57,260 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/100e85c16d67411d8fe30233be9c5931 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/100e85c16d67411d8fe30233be9c5931 2023-04-27 03:55:57,261 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/513df438a72649538dbfee7f48d8c5df to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/513df438a72649538dbfee7f48d8c5df 2023-04-27 03:55:57,261 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/3323f6a7c03c4d808dfd97ef2a4ebff2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/3323f6a7c03c4d808dfd97ef2a4ebff2 2023-04-27 03:55:57,262 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/b60f0a3743ae4d689c73f45d5b6d559e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/b60f0a3743ae4d689c73f45d5b6d559e 2023-04-27 03:55:57,263 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/28b838fcd47148d38a414739692b645e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/28b838fcd47148d38a414739692b645e 2023-04-27 03:55:57,263 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/e97158d2b8ce4b1ea4b05d396a64a1a2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/e97158d2b8ce4b1ea4b05d396a64a1a2 2023-04-27 03:55:57,264 DEBUG [StoreCloser-TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/7cc58619c9aa4a4dba0e764dba799aa9 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/7cc58619c9aa4a4dba0e764dba799aa9 2023-04-27 03:55:57,268 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/recovered.edits/526.seqid, newMaxSeqId=526, maxSeqId=1 2023-04-27 03:55:57,269 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4. 2023-04-27 03:55:57,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 98cbc3f1a78abce9e99d0349272cd6a4: 2023-04-27 03:55:57,270 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:57,271 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=98cbc3f1a78abce9e99d0349272cd6a4, regionState=CLOSED 2023-04-27 03:55:57,271 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567757271"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682567757271"}]},"ts":"1682567757271"} 2023-04-27 03:55:57,273 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-27 03:55:57,273 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 98cbc3f1a78abce9e99d0349272cd6a4, server=jenkins-hbase4.apache.org,40885,1682567618305 in 2.2840 sec 2023-04-27 03:55:57,274 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-27 03:55:57,274 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=98cbc3f1a78abce9e99d0349272cd6a4, UNASSIGN in 2.2880 sec 2023-04-27 03:55:57,274 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567757274"}]},"ts":"1682567757274"} 2023-04-27 03:55:57,275 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-27 03:55:57,278 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-27 03:55:57,279 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.3020 sec 2023-04-27 03:55:59,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-27 03:55:59,086 INFO [Listener at localhost/33933] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-27 03:55:59,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-27 03:55:59,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:55:59,089 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:55:59,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-27 03:55:59,089 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:55:59,093 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:59,096 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A, FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B, FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C, FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/recovered.edits] 2023-04-27 03:55:59,099 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/46ddd1f56279400d8785c18b33a8e975 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/46ddd1f56279400d8785c18b33a8e975 2023-04-27 03:55:59,100 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/7cd32992135e4674b11e3932c4a7f070 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/7cd32992135e4674b11e3932c4a7f070 2023-04-27 03:55:59,101 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/b95ee5ba6ebf49f492b0b6abdc024b43 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/b95ee5ba6ebf49f492b0b6abdc024b43 2023-04-27 03:55:59,102 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/fc364bba45b54c49a9773e96075e4f9a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/A/fc364bba45b54c49a9773e96075e4f9a 2023-04-27 03:55:59,104 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/749c8208a61d4064a557ad5e7bb4599d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/749c8208a61d4064a557ad5e7bb4599d 2023-04-27 03:55:59,105 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/8835cdad58394ea7aff9c5bdc82f1348 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/8835cdad58394ea7aff9c5bdc82f1348 2023-04-27 03:55:59,105 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/c36cce9ef173436eb37e4a30e741cb92 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/c36cce9ef173436eb37e4a30e741cb92 2023-04-27 03:55:59,106 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/ca88aeaa8cbb4ca3baf8d1c1824700f8 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/B/ca88aeaa8cbb4ca3baf8d1c1824700f8 2023-04-27 03:55:59,108 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/0777ed50cfbb45a4829f6a9ad0c0479a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/0777ed50cfbb45a4829f6a9ad0c0479a 2023-04-27 03:55:59,109 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/56f103b7bb114c22bacc4c56c0acc451 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/56f103b7bb114c22bacc4c56c0acc451 2023-04-27 03:55:59,109 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/db4a7728d4af446ab1aa2ad85961b2bd to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/db4a7728d4af446ab1aa2ad85961b2bd 2023-04-27 03:55:59,110 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/fa1e4305785c4250abee9c9682156057 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/C/fa1e4305785c4250abee9c9682156057 2023-04-27 03:55:59,112 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/recovered.edits/526.seqid to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4/recovered.edits/526.seqid 2023-04-27 03:55:59,113 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/98cbc3f1a78abce9e99d0349272cd6a4 2023-04-27 03:55:59,113 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-27 03:55:59,114 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:55:59,115 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-27 03:55:59,117 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-27 03:55:59,117 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:55:59,117 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-27 03:55:59,118 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682567759117"}]},"ts":"9223372036854775807"} 2023-04-27 03:55:59,119 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-27 03:55:59,119 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 98cbc3f1a78abce9e99d0349272cd6a4, NAME => 'TestAcidGuarantees,,1682567732410.98cbc3f1a78abce9e99d0349272cd6a4.', STARTKEY => '', ENDKEY => ''}] 2023-04-27 03:55:59,119 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-27 03:55:59,119 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682567759119"}]},"ts":"9223372036854775807"} 2023-04-27 03:55:59,120 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-27 03:55:59,122 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:55:59,122 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 35 msec 2023-04-27 03:55:59,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-27 03:55:59,190 INFO [Listener at localhost/33933] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-27 03:55:59,201 INFO [Listener at localhost/33933] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=228 (was 224) - Thread LEAK? -, OpenFileDescriptor=566 (was 531) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=371 (was 306) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=12287 (was 11875) - AvailableMemoryMB LEAK? - 2023-04-27 03:55:59,213 INFO [Listener at localhost/33933] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228, OpenFileDescriptor=566, MaxFileDescriptor=60000, SystemLoadAverage=371, ProcessCount=170, AvailableMemoryMB=12288 2023-04-27 03:55:59,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-27 03:55:59,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-27 03:55:59,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-27 03:55:59,217 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-27 03:55:59,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-27 03:55:59,218 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-27 03:55:59,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-27 03:55:59,219 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:55:59,219 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231 empty. 2023-04-27 03:55:59,220 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:55:59,220 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-27 03:55:59,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-27 03:55:59,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-27 03:55:59,626 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-27 03:55:59,627 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 1c25ca7c0495563171ae7ccdf0350231, NAME => 'TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp 2023-04-27 03:55:59,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-27 03:56:00,033 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 03:56:00,033 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 1c25ca7c0495563171ae7ccdf0350231, disabling compactions & flushes 2023-04-27 03:56:00,033 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:00,033 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:00,033 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. after waiting 0 ms 2023-04-27 03:56:00,033 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:00,033 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:00,033 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:00,035 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-27 03:56:00,036 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567760035"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682567760035"}]},"ts":"1682567760035"} 2023-04-27 03:56:00,036 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-27 03:56:00,037 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-27 03:56:00,037 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567760037"}]},"ts":"1682567760037"} 2023-04-27 03:56:00,038 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-27 03:56:00,041 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1c25ca7c0495563171ae7ccdf0350231, ASSIGN}] 2023-04-27 03:56:00,042 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1c25ca7c0495563171ae7ccdf0350231, ASSIGN 2023-04-27 03:56:00,043 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=1c25ca7c0495563171ae7ccdf0350231, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40885,1682567618305; forceNewPlan=false, retain=false 2023-04-27 03:56:00,194 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=1c25ca7c0495563171ae7ccdf0350231, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:00,194 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567760194"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682567760194"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682567760194"}]},"ts":"1682567760194"} 2023-04-27 03:56:00,196 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305}] 2023-04-27 03:56:00,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-27 03:56:00,350 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:00,350 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1c25ca7c0495563171ae7ccdf0350231, NAME => 'TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.', STARTKEY => '', ENDKEY => ''} 2023-04-27 03:56:00,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:00,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 03:56:00,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:00,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:00,352 INFO [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:00,353 DEBUG [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A 2023-04-27 03:56:00,353 DEBUG [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A 2023-04-27 03:56:00,353 INFO [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:56:00,354 INFO [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1c25ca7c0495563171ae7ccdf0350231 columnFamilyName A 2023-04-27 03:56:00,354 INFO [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] regionserver.HStore(310): Store=1c25ca7c0495563171ae7ccdf0350231/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:56:00,354 INFO [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:00,355 DEBUG [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B 2023-04-27 03:56:00,355 DEBUG [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B 2023-04-27 03:56:00,355 INFO [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:56:00,355 INFO [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1c25ca7c0495563171ae7ccdf0350231 columnFamilyName B 2023-04-27 03:56:00,356 INFO [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] regionserver.HStore(310): Store=1c25ca7c0495563171ae7ccdf0350231/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:56:00,356 INFO [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:00,357 DEBUG [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C 2023-04-27 03:56:00,357 DEBUG [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C 2023-04-27 03:56:00,357 INFO [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:56:00,357 INFO [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1c25ca7c0495563171ae7ccdf0350231 columnFamilyName C 2023-04-27 03:56:00,358 INFO [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] regionserver.HStore(310): Store=1c25ca7c0495563171ae7ccdf0350231/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:56:00,358 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:00,358 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:00,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:00,360 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-27 03:56:00,361 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:00,363 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-27 03:56:00,363 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1c25ca7c0495563171ae7ccdf0350231; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72519363, jitterRate=0.08062271773815155}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-27 03:56:00,363 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:00,364 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., pid=56, masterSystemTime=1682567760347 2023-04-27 03:56:00,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:00,365 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:00,365 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=1c25ca7c0495563171ae7ccdf0350231, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:00,365 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567760365"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682567760365"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682567760365"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682567760365"}]},"ts":"1682567760365"} 2023-04-27 03:56:00,367 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-27 03:56:00,367 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 in 170 msec 2023-04-27 03:56:00,368 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-27 03:56:00,368 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1c25ca7c0495563171ae7ccdf0350231, ASSIGN in 326 msec 2023-04-27 03:56:00,369 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-27 03:56:00,369 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567760369"}]},"ts":"1682567760369"} 2023-04-27 03:56:00,369 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-27 03:56:00,371 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-27 03:56:00,372 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1560 sec 2023-04-27 03:56:01,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-27 03:56:01,323 INFO [Listener at localhost/33933] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-27 03:56:01,323 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x45878790 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:56:01,328 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e3503b4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:56:01,329 DEBUG [Listener at localhost/33933] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:56:01,330 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48428, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:56:01,331 DEBUG [Listener at localhost/33933] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-27 03:56:01,333 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43278, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-27 03:56:01,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-27 03:56:01,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-27 03:56:01,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-27 03:56:01,744 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-27 03:56:01,745 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-27 03:56:01,745 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-27 03:56:01,747 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-27 03:56:01,749 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1c25ca7c0495563171ae7ccdf0350231, REOPEN/MOVE}] 2023-04-27 03:56:01,749 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1c25ca7c0495563171ae7ccdf0350231, REOPEN/MOVE 2023-04-27 03:56:01,750 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=1c25ca7c0495563171ae7ccdf0350231, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:01,750 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567761750"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682567761750"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682567761750"}]},"ts":"1682567761750"} 2023-04-27 03:56:01,751 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305}] 2023-04-27 03:56:01,903 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:01,904 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1c25ca7c0495563171ae7ccdf0350231, disabling compactions & flushes 2023-04-27 03:56:01,904 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:01,904 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:01,904 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. after waiting 0 ms 2023-04-27 03:56:01,904 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:01,907 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-27 03:56:01,907 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:01,907 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:01,907 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 1c25ca7c0495563171ae7ccdf0350231 to self. 2023-04-27 03:56:01,908 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:01,909 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=1c25ca7c0495563171ae7ccdf0350231, regionState=CLOSED 2023-04-27 03:56:01,909 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567761909"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682567761909"}]},"ts":"1682567761909"} 2023-04-27 03:56:01,912 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-27 03:56:01,912 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 in 159 msec 2023-04-27 03:56:01,913 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=1c25ca7c0495563171ae7ccdf0350231, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,40885,1682567618305; forceNewPlan=false, retain=true 2023-04-27 03:56:02,063 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=1c25ca7c0495563171ae7ccdf0350231, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,063 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567762063"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682567762063"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682567762063"}]},"ts":"1682567762063"} 2023-04-27 03:56:02,064 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305}] 2023-04-27 03:56:02,218 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:02,218 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1c25ca7c0495563171ae7ccdf0350231, NAME => 'TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.', STARTKEY => '', ENDKEY => ''} 2023-04-27 03:56:02,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:02,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-27 03:56:02,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:02,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:02,220 INFO [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:02,220 DEBUG [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A 2023-04-27 03:56:02,221 DEBUG [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A 2023-04-27 03:56:02,221 INFO [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:56:02,221 INFO [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1c25ca7c0495563171ae7ccdf0350231 columnFamilyName A 2023-04-27 03:56:02,222 INFO [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] regionserver.HStore(310): Store=1c25ca7c0495563171ae7ccdf0350231/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:56:02,222 INFO [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:02,223 DEBUG [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B 2023-04-27 03:56:02,223 DEBUG [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B 2023-04-27 03:56:02,223 INFO [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:56:02,223 INFO [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1c25ca7c0495563171ae7ccdf0350231 columnFamilyName B 2023-04-27 03:56:02,223 INFO [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] regionserver.HStore(310): Store=1c25ca7c0495563171ae7ccdf0350231/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:56:02,223 INFO [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:02,224 DEBUG [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C 2023-04-27 03:56:02,224 DEBUG [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C 2023-04-27 03:56:02,224 INFO [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-27 03:56:02,224 INFO [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1c25ca7c0495563171ae7ccdf0350231 columnFamilyName C 2023-04-27 03:56:02,225 INFO [StoreOpener-1c25ca7c0495563171ae7ccdf0350231-1] regionserver.HStore(310): Store=1c25ca7c0495563171ae7ccdf0350231/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-27 03:56:02,225 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:02,225 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:02,226 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:02,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-27 03:56:02,228 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:02,229 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1c25ca7c0495563171ae7ccdf0350231; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63281733, jitterRate=-0.057028695940971375}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-27 03:56:02,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:02,229 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., pid=61, masterSystemTime=1682567762216 2023-04-27 03:56:02,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:02,230 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:02,231 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=1c25ca7c0495563171ae7ccdf0350231, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,231 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567762231"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682567762231"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682567762231"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682567762231"}]},"ts":"1682567762231"} 2023-04-27 03:56:02,233 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-27 03:56:02,233 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 in 168 msec 2023-04-27 03:56:02,234 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-27 03:56:02,234 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1c25ca7c0495563171ae7ccdf0350231, REOPEN/MOVE in 484 msec 2023-04-27 03:56:02,235 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-27 03:56:02,235 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 487 msec 2023-04-27 03:56:02,236 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 900 msec 2023-04-27 03:56:02,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-27 03:56:02,237 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x1644730f to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:56:02,243 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f06dd50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:56:02,243 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x6af74c0e to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:56:02,249 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9e8654, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:56:02,250 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x1d759aab to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:56:02,254 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4db22237, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:56:02,254 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x39241fe3 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:56:02,259 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8b57aae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:56:02,260 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x1fbb5b46 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:56:02,263 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6799455, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:56:02,263 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x66c25427 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:56:02,268 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c53bf96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:56:02,269 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x4f4eab6d to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:56:02,272 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b59bdcf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:56:02,273 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x07d997c6 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:56:02,281 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d765193, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:56:02,282 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x7993d99e to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:56:02,285 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32bf16cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:56:02,285 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(139): Connect 0x4fae3ed2 to 127.0.0.1:55020 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-27 03:56:02,291 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28164190, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-27 03:56:02,305 DEBUG [hconnection-0x3e0f6201-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:56:02,305 DEBUG [hconnection-0x68c51a56-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:56:02,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:02,308 DEBUG [hconnection-0x23c066a9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:56:02,308 DEBUG [hconnection-0x49baa796-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:56:02,310 DEBUG [hconnection-0x19d075aa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:56:02,310 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48464, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:56:02,310 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48458, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:56:02,310 DEBUG [hconnection-0x61a030e3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:56:02,310 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48444, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:56:02,311 DEBUG [hconnection-0x4795d546-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:56:02,310 DEBUG [hconnection-0x29ef5a04-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:56:02,310 DEBUG [hconnection-0x3077f2df-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:56:02,312 DEBUG [hconnection-0xcbf44e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-27 03:56:02,312 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48474, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:56:02,312 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48488, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:56:02,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:02,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:02,313 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48502, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:56:02,313 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:02,313 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:02,313 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:02,313 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:02,320 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,321 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,321 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48508, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:56:02,321 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48522, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:56:02,321 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48534, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:56:02,321 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48532, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-27 03:56:02,323 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:02,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:02,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:02,324 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,324 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,324 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:02,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:02,331 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-27 03:56:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,331 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:02,332 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:02,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:02,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:02,332 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:02,332 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:02,332 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:02,332 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:02,332 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:02,332 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:02,332 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:02,332 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:02,344 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:02,344 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:02,344 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:02,344 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:02,344 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:02,344 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:02,344 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:02,344 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:02,344 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,344 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:02,344 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:02,346 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,346 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:02,347 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,347 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:02,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:02,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:02,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:02,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:02,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:02,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48502 deadline: 1682567822345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:02,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567822345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48474 deadline: 1682567822348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:02,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16aba266[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:02,349 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:02,350 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:02,351 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,351 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,351 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:02,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567822349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,351 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:02,351 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:02,351 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:02,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:02,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567822353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,354 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:02,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:02,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:02,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:02,354 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:02,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:02,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:02,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:02,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:02,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:02,357 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:02,357 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:02,357 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:02,357 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:02,357 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:02,359 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,359 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:02,359 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:02,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:02,359 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:02,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,361 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:02,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,368 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:02,368 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:02,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:02,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:02,369 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:02,369 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,369 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:02,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:02,369 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,369 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,369 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,369 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,369 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,369 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:02,369 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:02,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:02,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:02,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:02,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:02,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48502 deadline: 1682567822450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:02,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567822450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:02,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567822454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:02,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48474 deadline: 1682567822454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:02,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567822454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,469 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:02,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:02,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:02,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:02,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:02,476 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:02,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:02,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:02,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:02,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,479 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:02,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:02,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:02,479 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,479 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,479 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:02,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:02,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:02,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:02,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:02,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:02,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:02,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:02,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:02,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:02,481 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:02,481 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:02,481 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:02,481 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:02,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:02,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:02,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:02,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:02,483 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,483 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:02,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:02,483 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,483 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:02,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:02,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:02,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:02,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:02,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40c3af2b[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:02,485 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:02,485 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,486 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,486 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:02,486 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:02,486 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:02,486 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,486 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:02,488 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:02,488 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:02,488 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:02,488 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:02,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:02,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:02,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:02,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:02,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:02,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:02,490 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:02,490 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:02,490 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:02,490 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:02,490 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:02,494 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,494 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,494 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:02,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:02,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:02,494 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:02,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:02,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:02,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:02,495 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:02,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,499 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,499 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,499 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,499 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:02,499 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:02,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:02,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:02,499 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,499 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:02,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:02,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:02,499 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,499 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,499 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,499 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:02,500 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:02,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:02,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:02,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:02,600 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:02,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:02,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:02,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:02,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:02,603 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:02,603 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:02,603 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:02,603 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:02,603 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,604 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,605 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,605 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:02,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:02,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:02,605 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,605 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:02,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:02,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:02,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:02,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:02,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:02,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:02,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:02,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:02,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:02,606 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:02,607 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:02,607 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:02,607 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:02,607 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:02,607 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:02,607 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,607 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:02,607 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:02,608 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,608 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:02,608 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,608 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:02,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:02,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:02,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:02,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73b87339[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:02,610 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:02,610 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,612 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:02,612 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:02,612 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:02,612 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,612 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:02,613 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:02,613 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:02,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,613 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:02,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:02,616 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:02,616 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:02,616 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:02,616 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:02,616 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:02,618 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,618 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:02,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,618 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:02,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:02,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:02,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:02,619 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:02,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:02,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,620 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:02,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:02,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:02,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:02,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:02,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,623 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,623 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,623 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,623 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:02,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:02,623 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:02,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:02,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:02,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:02,623 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,623 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,623 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:02,623 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:02,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:02,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48502 deadline: 1682567822652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:02,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567822653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:02,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:02,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567822657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567822657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:02,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48474 deadline: 1682567822657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,723 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:02,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:02,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:02,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:02,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:02,727 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:02,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:02,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:02,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:02,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,729 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:02,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:02,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:02,730 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,730 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:02,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:02,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,731 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:02,731 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:02,731 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:02,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:02,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:02,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:02,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:02,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:02,732 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:02,732 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:02,732 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:02,732 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:02,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:02,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:02,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:02,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:02,734 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,734 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:02,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:02,734 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:02,734 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:02,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:02,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:02,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:02,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@86a36a[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:02,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:02,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,737 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,737 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:02,737 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:02,737 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:02,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:02,738 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:02,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:02,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:02,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:02,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:02,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:02,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:02,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:02,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:02,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:02,741 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:02,741 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:02,741 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:02,741 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:02,741 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:02,744 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,744 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:02,744 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:02,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:02,745 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:02,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,746 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:02,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:02,749 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:02,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:02,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:02,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:02,749 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:02,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:02,749 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:02,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:02,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:02,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:02,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:02,785 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427be24d603aedc4001ae38e9b95f2cf820 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427be24d603aedc4001ae38e9b95f2cf820 2023-04-27 03:56:02,809 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-27 03:56:02,849 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:02,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:02,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:02,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:02,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:02,864 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:02,864 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:02,864 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:02,864 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:02,864 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,865 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,868 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,868 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:02,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:02,869 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,869 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:02,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:02,870 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:02,870 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:02,870 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:02,870 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:02,870 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:02,884 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:02,884 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:02,884 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:02,884 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:02,884 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:02,884 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:02,884 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:02,884 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:02,884 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,884 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:02,884 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:02,887 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,887 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:02,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:02,888 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,888 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:02,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:02,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:02,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:02,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:02,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b75998e[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:02,889 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:02,889 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,890 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,891 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:02,891 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:02,891 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:02,891 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,891 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:02,893 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:02,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:02,899 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:02,900 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:02,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:02,900 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:02,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:02,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:02,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:02,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:02,903 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:02,903 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:02,903 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:02,903 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:02,903 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:02,904 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,905 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,905 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:02,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:02,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:02,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:02,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:02,905 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:02,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:02,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,907 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:02,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:02,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,910 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,911 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:02,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:02,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:02,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:02,911 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:02,911 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:02,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:02,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:02,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:02,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:02,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:02,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:02,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:02,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:02,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:02,955 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:02,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48502 deadline: 1682567822954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:02,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567822955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:02,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567822960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:02,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567822960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:02,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:02,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48474 deadline: 1682567822961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,011 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:03,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:03,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:03,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:03,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:03,015 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:03,015 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:03,015 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:03,015 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:03,016 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,016 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,017 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,017 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:03,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:03,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:03,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,018 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,018 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:03,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:03,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:03,019 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:03,019 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:03,019 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:03,019 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:03,019 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:03,020 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:03,020 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:03,020 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:03,020 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:03,020 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:03,020 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:03,020 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:03,020 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:03,020 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,020 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:03,020 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:03,022 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,022 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:03,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:03,022 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,022 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:03,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:03,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:03,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:03,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:03,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1612cd3c[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:03,024 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:03,024 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,026 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,026 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:03,026 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:03,026 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:03,026 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,026 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:03,027 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:03,028 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:03,028 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:03,028 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:03,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:03,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:03,032 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:03,032 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:03,032 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:03,032 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:03,032 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:03,034 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,034 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:03,034 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:03,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:03,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:03,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:03,034 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:03,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:03,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,037 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,040 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,040 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,040 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,040 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,040 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:03,040 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:03,040 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:03,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:03,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:03,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:03,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:03,040 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,040 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,040 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:03,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:03,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:03,040 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:03,041 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:03,141 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:03,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:03,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:03,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:03,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:03,144 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:03,144 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:03,144 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:03,144 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:03,144 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,145 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,147 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,147 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:03,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:03,147 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,147 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:03,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:03,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:03,149 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:03,149 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:03,149 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:03,149 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:03,149 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:03,149 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:03,149 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:03,149 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:03,149 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:03,149 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:03,149 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:03,149 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,149 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:03,149 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:03,151 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:03,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:03,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:03,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:03,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2603557c[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:03,152 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:03,153 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,154 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,154 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:03,154 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:03,154 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:03,154 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,154 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:03,155 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,155 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:03,156 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:03,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:03,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:03,156 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:03,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:03,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:03,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:03,158 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:03,158 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:03,158 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:03,158 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:03,158 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:03,166 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,166 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,166 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:03,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:03,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:03,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:03,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:03,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:03,166 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:03,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,168 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:03,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,170 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,170 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,170 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:03,170 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:03,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:03,170 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:03,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:03,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:03,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:03,171 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:03,171 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:03,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:03,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:03,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:03,170 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,171 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,171 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,171 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,171 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,190 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/acfbeada10d34d91959f1b857d5a5f33 2023-04-27 03:56:03,271 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:03,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:03,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:03,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:03,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:03,274 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:03,274 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:03,274 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:03,274 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:03,274 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,275 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,276 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,276 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:03,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:03,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:03,276 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,276 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:03,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:03,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:03,277 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:03,277 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:03,277 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:03,277 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:03,277 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:03,277 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:03,277 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:03,277 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:03,278 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:03,278 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:03,278 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:03,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:03,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:03,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:03,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:03,284 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,284 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:03,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:03,285 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,285 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:03,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:03,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:03,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:03,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:03,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@132b3f25[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:03,286 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:03,286 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,288 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,288 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:03,288 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:03,288 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:03,288 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,288 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:03,289 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:03,289 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,289 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:03,289 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:03,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:03,292 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:03,292 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:03,292 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:03,292 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:03,292 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:03,293 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,293 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,293 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:03,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:03,294 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,296 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,299 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,299 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:03,299 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:03,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:03,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:03,299 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,299 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,299 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,299 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,299 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,299 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:03,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:03,299 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:03,299 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:03,300 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:03,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:03,400 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:03,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:03,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:03,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:03,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:03,403 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:03,403 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:03,403 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:03,403 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:03,404 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,404 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,405 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,405 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:03,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:03,405 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,406 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:03,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:03,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,406 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:03,406 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:03,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:03,406 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:03,407 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:03,407 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:03,407 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:03,407 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:03,407 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:03,407 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:03,407 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:03,407 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:03,407 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:03,407 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:03,407 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,407 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:03,407 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:03,409 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,409 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:03,409 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,410 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:03,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:03,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74f57d27[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:03,411 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:03,411 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,412 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,412 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:03,412 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,413 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:03,413 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:03,413 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:03,414 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:03,415 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:03,415 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:03,415 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:03,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:03,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:03,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:03,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:03,417 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:03,418 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:03,418 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:03,418 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:03,418 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:03,419 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:03,419 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,419 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:03,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:03,420 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,421 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,423 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:03,423 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:03,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:03,423 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:03,423 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,423 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:03,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:03,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:03,423 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,423 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,423 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,423 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,423 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:03,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:03,424 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:03,424 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:03,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:03,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567823457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:03,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48502 deadline: 1682567823459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:03,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567823462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:03,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48474 deadline: 1682567823463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,466 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:03,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567823465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,524 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:03,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:03,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:03,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:03,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:03,527 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:03,527 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:03,528 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:03,528 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:03,528 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,528 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,529 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:03,529 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:03,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:03,530 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,530 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:03,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:03,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:03,531 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:03,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:03,531 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:03,531 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:03,531 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:03,531 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:03,531 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:03,531 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:03,531 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:03,532 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:03,532 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:03,532 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:03,532 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:03,532 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,532 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:03,532 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:03,534 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,534 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:03,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:03,535 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,535 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:03,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:03,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:03,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:03,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:03,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@740e35a8[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:03,537 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:03,537 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,538 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,538 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:03,538 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:03,538 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:03,538 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,538 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:03,539 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:03,539 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:03,539 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:03,540 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:03,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:03,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:03,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:03,543 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:03,543 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:03,543 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:03,543 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:03,543 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:03,544 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,544 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:03,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,545 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:03,545 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,546 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:03,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:03,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,550 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:03,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:03,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:03,551 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,551 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,551 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,551 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,551 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:03,551 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:03,551 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,551 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:03,551 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:03,551 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:03,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:03,551 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:03,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:03,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:03,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:03,601 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/9b056c567a6e4ef4b9d7affb202c5666 2023-04-27 03:56:03,613 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/dcbab4ee2e964367bce2defd9eb530dd 2023-04-27 03:56:03,617 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/acfbeada10d34d91959f1b857d5a5f33 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/acfbeada10d34d91959f1b857d5a5f33 2023-04-27 03:56:03,621 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/acfbeada10d34d91959f1b857d5a5f33, entries=150, sequenceid=17, filesize=25.1 K 2023-04-27 03:56:03,622 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/9b056c567a6e4ef4b9d7affb202c5666 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/9b056c567a6e4ef4b9d7affb202c5666 2023-04-27 03:56:03,625 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/9b056c567a6e4ef4b9d7affb202c5666, entries=150, sequenceid=17, filesize=11.6 K 2023-04-27 03:56:03,625 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/dcbab4ee2e964367bce2defd9eb530dd as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/dcbab4ee2e964367bce2defd9eb530dd 2023-04-27 03:56:03,629 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/dcbab4ee2e964367bce2defd9eb530dd, entries=150, sequenceid=17, filesize=11.6 K 2023-04-27 03:56:03,629 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 1c25ca7c0495563171ae7ccdf0350231 in 1299ms, sequenceid=17, compaction requested=false 2023-04-27 03:56:03,629 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-27 03:56:03,630 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:03,651 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:03,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:03,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:03,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:03,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:03,654 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:03,654 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:03,654 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:03,654 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:03,655 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,655 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,656 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,656 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:03,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:03,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:03,656 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:03,656 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:03,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:03,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:03,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:03,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:03,657 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:03,657 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:03,657 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:03,658 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:03,658 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:03,658 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:03,658 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:03,658 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:03,658 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:03,658 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-27 03:56:03,658 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:03,658 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:03,658 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:03,658 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:03,658 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:03,658 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:03,687 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427bf91b69429af4993ad424a923886e925 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427bf91b69429af4993ad424a923886e925 2023-04-27 03:56:04,092 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/969653295d8e41039b0d3dc9085acd41 2023-04-27 03:56:04,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:04,463 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:04,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:04,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567824471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:04,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48502 deadline: 1682567824473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:04,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:04,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567824474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567824474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:04,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48474 deadline: 1682567824475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,503 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/0666067f538c41c9b5c2b33cc5f43751 2023-04-27 03:56:04,514 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/4897bb40512d499e90561b8c19fd87ef 2023-04-27 03:56:04,519 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/969653295d8e41039b0d3dc9085acd41 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/969653295d8e41039b0d3dc9085acd41 2023-04-27 03:56:04,523 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/969653295d8e41039b0d3dc9085acd41, entries=150, sequenceid=36, filesize=25.1 K 2023-04-27 03:56:04,524 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/0666067f538c41c9b5c2b33cc5f43751 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/0666067f538c41c9b5c2b33cc5f43751 2023-04-27 03:56:04,527 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/0666067f538c41c9b5c2b33cc5f43751, entries=150, sequenceid=36, filesize=11.6 K 2023-04-27 03:56:04,528 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/4897bb40512d499e90561b8c19fd87ef as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/4897bb40512d499e90561b8c19fd87ef 2023-04-27 03:56:04,530 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/4897bb40512d499e90561b8c19fd87ef, entries=150, sequenceid=36, filesize=11.6 K 2023-04-27 03:56:04,531 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 1c25ca7c0495563171ae7ccdf0350231 in 873ms, sequenceid=36, compaction requested=false 2023-04-27 03:56:04,532 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:04,532 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:04,532 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:04,532 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:04,533 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,533 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:04,533 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:04,534 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,534 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:04,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:04,534 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,534 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:04,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:04,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:04,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:04,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:04,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:04,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@489a26e0[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:04,536 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:04,536 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,537 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,537 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:04,537 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:04,537 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:04,537 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,537 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:04,542 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,542 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:04,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:04,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:04,542 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:04,543 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:04,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:04,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:04,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:04,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:04,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:04,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:04,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:04,545 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:04,545 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:04,545 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:04,545 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:04,545 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:04,546 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,546 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,546 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:04,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:04,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:04,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:04,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:04,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:04,547 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:04,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:04,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:04,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,549 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:04,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:04,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,551 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,551 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:04,551 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:04,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:04,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:04,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:04,551 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:04,551 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:04,551 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:04,551 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,551 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,551 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,551 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:04,551 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:04,551 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:04,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:04,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:04,578 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-27 03:56:04,578 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:04,578 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:04,578 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:04,578 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:04,578 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:04,578 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:04,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:04,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567824590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:04,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:04,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48502 deadline: 1682567824591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567824591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:04,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567824592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,651 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:04,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:04,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:04,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:04,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:04,654 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:04,654 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:04,654 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:04,655 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:04,655 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,655 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:04,656 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:04,656 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:04,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:04,656 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,656 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:04,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:04,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:04,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:04,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:04,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:04,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:04,658 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:04,658 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:04,658 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:04,658 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:04,658 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:04,658 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:04,658 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:04,658 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:04,658 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:04,658 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:04,658 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:04,658 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,658 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:04,658 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:04,660 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,660 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:04,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:04,660 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,661 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:04,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:04,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:04,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:04,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:04,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:04,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@282eac1[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:04,662 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:04,662 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,663 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,663 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:04,663 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:04,663 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:04,663 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,663 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:04,664 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:04,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:04,664 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:04,665 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:04,665 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:04,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:04,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:04,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:04,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:04,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:04,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:04,667 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:04,667 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:04,667 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:04,667 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:04,667 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:04,669 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,669 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:04,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,669 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:04,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:04,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:04,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:04,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:04,669 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:04,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:04,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:04,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,670 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:04,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:04,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,674 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,674 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:04,674 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:04,674 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:04,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:04,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:04,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:04,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:04,674 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:04,674 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:04,675 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:04,675 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,675 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:04,675 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,675 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,675 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:04,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:04,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:04,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567824694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:04,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567824695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:04,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48502 deadline: 1682567824695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:04,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567824695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,775 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:04,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:04,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:04,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:04,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:04,778 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:04,778 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:04,778 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:04,778 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:04,779 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,779 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:04,780 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,780 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:04,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:04,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:04,780 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,780 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:04,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:04,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:04,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:04,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:04,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:04,781 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:04,781 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:04,781 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:04,781 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:04,782 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:04,782 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:04,782 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:04,782 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:04,782 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:04,782 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:04,782 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:04,782 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,782 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:04,782 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:04,783 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,783 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:04,784 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,784 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:04,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:04,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:04,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dc1a3f3[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:04,785 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:04,785 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,787 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,787 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:04,787 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:04,787 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:04,787 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,787 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:04,789 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:04,789 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:04,789 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,789 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:04,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:04,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:04,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:04,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:04,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:04,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:04,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:04,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:04,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:04,792 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:04,792 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:04,792 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:04,792 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:04,792 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:04,793 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,793 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:04,793 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:04,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:04,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:04,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:04,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:04,794 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:04,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:04,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:04,795 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:04,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:04,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,797 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:04,797 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:04,797 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:04,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:04,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:04,797 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:04,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:04,798 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:04,798 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:04,798 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,798 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,798 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,798 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,798 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:04,798 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:04,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:04,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:04,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:04,898 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:04,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:04,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:04,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:04,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567824898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:04,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:04,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:04,901 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:04,901 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:04,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567824899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,901 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:04,901 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:04,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:04,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567824899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,902 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,902 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:04,903 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,903 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:04,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:04,903 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,903 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:04,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:04,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:04,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:04,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:04,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:04,904 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:04,904 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:04,904 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:04,904 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:04,904 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:04,904 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:04,905 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:04,905 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:04,905 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:04,905 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:04,905 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:04,905 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,905 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:04,905 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:04,907 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,907 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:04,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:04,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:04,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48502 deadline: 1682567824905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:04,908 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,908 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:04,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:04,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:04,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:04,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:04,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58676e96[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:04,909 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:04,909 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,910 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,910 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:04,910 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:04,910 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:04,910 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,910 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:04,911 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:04,911 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:04,911 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:04,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:04,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:04,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:04,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:04,914 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:04,914 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:04,914 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:04,914 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:04,914 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:04,918 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,918 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,918 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:04,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:04,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:04,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:04,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:04,919 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:04,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:04,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:04,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:04,920 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:04,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:04,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,922 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,922 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:04,922 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:04,922 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:04,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:04,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:04,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:04,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:04,922 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:04,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:04,922 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:04,923 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:04,923 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:04,922 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:04,923 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,923 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:04,923 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:04,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:04,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:04,993 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304270f21f73eddae4d6cbae67e3a1c103780 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304270f21f73eddae4d6cbae67e3a1c103780 2023-04-27 03:56:04,997 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=52, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/6f12e5f9e8f24cc994b2cc8629ad9403 2023-04-27 03:56:05,023 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:05,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:05,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:05,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:05,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:05,026 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:05,026 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:05,026 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:05,026 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:05,026 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,026 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,027 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,027 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:05,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:05,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:05,027 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,027 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:05,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:05,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,028 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:05,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:05,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:05,029 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:05,029 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:05,029 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:05,029 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:05,029 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:05,029 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:05,029 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:05,029 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:05,029 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:05,029 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:05,029 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:05,029 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,029 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:05,029 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:05,031 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,031 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:05,031 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,031 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:05,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:05,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4153fe5c[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:05,033 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:05,033 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,035 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,035 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:05,035 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:05,035 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,035 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:05,035 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:05,036 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,036 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:05,036 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:05,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:05,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:05,036 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:05,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:05,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:05,038 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:05,039 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:05,039 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:05,039 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:05,039 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:05,040 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:05,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:05,041 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,041 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:05,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:05,041 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:05,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:05,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:05,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:05,042 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,044 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,044 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,044 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,044 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:05,045 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:05,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:05,045 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:05,045 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,045 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:05,045 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:05,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:05,045 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:05,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:05,045 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:05,045 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:05,145 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:05,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:05,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:05,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:05,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:05,148 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:05,149 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:05,149 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:05,149 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:05,149 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,149 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,151 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:05,151 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:05,151 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,151 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:05,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:05,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:05,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:05,152 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:05,152 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:05,152 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:05,153 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:05,153 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:05,153 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:05,153 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:05,153 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:05,153 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:05,153 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:05,153 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:05,153 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,153 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:05,153 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:05,154 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,154 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:05,155 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,155 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:05,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:05,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5412c055[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:05,156 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:05,156 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,157 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,157 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:05,158 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:05,158 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:05,158 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,158 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:05,159 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:05,159 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,159 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:05,159 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:05,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:05,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:05,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:05,161 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:05,161 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:05,161 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:05,161 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:05,161 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:05,164 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,164 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:05,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:05,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,164 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:05,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:05,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:05,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:05,164 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:05,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,166 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:05,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,168 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:05,168 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:05,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:05,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:05,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:05,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:05,168 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,168 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,168 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:05,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:05,168 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:05,168 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:05,168 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:05,168 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:05,168 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,168 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,168 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:05,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567825202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:05,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567825202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:05,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567825204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:05,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48502 deadline: 1682567825210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,268 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:05,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:05,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:05,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:05,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:05,272 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:05,272 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:05,272 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:05,272 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:05,272 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,273 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,274 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:05,274 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:05,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:05,274 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,274 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:05,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:05,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:05,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:05,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:05,275 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:05,276 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:05,276 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:05,276 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:05,276 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:05,276 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:05,276 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:05,276 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:05,276 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:05,276 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:05,276 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:05,276 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,276 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:05,276 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:05,278 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:05,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:05,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:05,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:05,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:05,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:05,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:05,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41c3f1f[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:05,280 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:05,280 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,282 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,282 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,282 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:05,282 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:05,283 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:05,283 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:05,284 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:05,284 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:05,284 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,284 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:05,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:05,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:05,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:05,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:05,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:05,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:05,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:05,287 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:05,287 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:05,287 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:05,287 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:05,287 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:05,288 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:05,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:05,289 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,289 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:05,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:05,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:05,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:05,289 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:05,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,292 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:05,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,302 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:05,302 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:05,302 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:05,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:05,302 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:05,302 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,302 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,302 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,302 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:05,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:05,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:05,302 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:05,303 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,303 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:05,303 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:05,403 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:05,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:05,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:05,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:05,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:05,406 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:05,406 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:05,406 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:05,406 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:05,407 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,407 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,408 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,408 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:05,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:05,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:05,408 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,408 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:05,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:05,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:05,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:05,409 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:05,409 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:05,409 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:05,409 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:05,409 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:05,410 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:05,410 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:05,410 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:05,410 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:05,410 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:05,410 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:05,410 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,410 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:05,410 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:05,411 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,411 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:05,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:05,412 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,412 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:05,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:05,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67ecf2d4[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:05,414 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:05,414 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,415 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,415 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:05,415 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:05,415 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:05,416 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,416 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:05,418 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:05,418 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:05,418 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:05,418 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/932deb8352264c9ca4e36d4c402f5e43 2023-04-27 03:56:05,418 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:05,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:05,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:05,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:05,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:05,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:05,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:05,423 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:05,423 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:05,423 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:05,423 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:05,423 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:05,425 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:05,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:05,425 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:05,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,426 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:05,426 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:05,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:05,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:05,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,427 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:05,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,430 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/c34472fdba6c428b8176053b2aedff7c 2023-04-27 03:56:05,430 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:05,430 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,430 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,430 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,430 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:05,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:05,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:05,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:05,430 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:05,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:05,430 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:05,431 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,431 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:05,431 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:05,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:05,431 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,431 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:05,434 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/6f12e5f9e8f24cc994b2cc8629ad9403 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/6f12e5f9e8f24cc994b2cc8629ad9403 2023-04-27 03:56:05,437 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/6f12e5f9e8f24cc994b2cc8629ad9403, entries=200, sequenceid=52, filesize=31.9 K 2023-04-27 03:56:05,438 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/932deb8352264c9ca4e36d4c402f5e43 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/932deb8352264c9ca4e36d4c402f5e43 2023-04-27 03:56:05,444 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/932deb8352264c9ca4e36d4c402f5e43, entries=150, sequenceid=52, filesize=11.6 K 2023-04-27 03:56:05,445 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/c34472fdba6c428b8176053b2aedff7c as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/c34472fdba6c428b8176053b2aedff7c 2023-04-27 03:56:05,449 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/c34472fdba6c428b8176053b2aedff7c, entries=150, sequenceid=52, filesize=11.6 K 2023-04-27 03:56:05,449 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 1c25ca7c0495563171ae7ccdf0350231 in 872ms, sequenceid=52, compaction requested=true 2023-04-27 03:56:05,449 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:05,449 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:05,449 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:56:05,450 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:56:05,449 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:05,450 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:05,450 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84178 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:56:05,450 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:56:05,450 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/A is initiating minor compaction (all files) 2023-04-27 03:56:05,450 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/B is initiating minor compaction (all files) 2023-04-27 03:56:05,450 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/A in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:05,450 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/B in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:05,451 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/acfbeada10d34d91959f1b857d5a5f33, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/969653295d8e41039b0d3dc9085acd41, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/6f12e5f9e8f24cc994b2cc8629ad9403] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=82.2 K 2023-04-27 03:56:05,451 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/9b056c567a6e4ef4b9d7affb202c5666, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/0666067f538c41c9b5c2b33cc5f43751, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/932deb8352264c9ca4e36d4c402f5e43] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=34.8 K 2023-04-27 03:56:05,451 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting acfbeada10d34d91959f1b857d5a5f33, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1682567762322 2023-04-27 03:56:05,451 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 9b056c567a6e4ef4b9d7affb202c5666, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1682567762322 2023-04-27 03:56:05,451 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 969653295d8e41039b0d3dc9085acd41, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1682567762341 2023-04-27 03:56:05,451 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 0666067f538c41c9b5c2b33cc5f43751, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1682567762341 2023-04-27 03:56:05,451 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 932deb8352264c9ca4e36d4c402f5e43, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1682567764472 2023-04-27 03:56:05,451 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 6f12e5f9e8f24cc994b2cc8629ad9403, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1682567764472 2023-04-27 03:56:05,463 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#B#compaction#524 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:05,467 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#A#compaction#525 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:05,531 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:05,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:05,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:05,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:05,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:05,535 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:05,535 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:05,535 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:05,535 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:05,535 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,535 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,536 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,536 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:05,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:05,537 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,537 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:05,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:05,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:05,538 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:05,538 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:05,538 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:05,538 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:05,538 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:05,538 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:05,538 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:05,538 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-27 03:56:05,539 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:05,539 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:05,539 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:05,539 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:05,539 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:05,539 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:05,553 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427340bb7073b5c49638ca9c504c50aea4d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427340bb7073b5c49638ca9c504c50aea4d 2023-04-27 03:56:05,559 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=68, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/7606d53b0e1a45c4a0dd8512512bb7cf 2023-04-27 03:56:05,568 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/02f10fd10b534878a63b0658211e6a38 2023-04-27 03:56:05,578 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/c5aadee936d84e4fb683fc3a791f6a36 2023-04-27 03:56:05,581 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/7606d53b0e1a45c4a0dd8512512bb7cf as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/7606d53b0e1a45c4a0dd8512512bb7cf 2023-04-27 03:56:05,585 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/7606d53b0e1a45c4a0dd8512512bb7cf, entries=150, sequenceid=68, filesize=25.1 K 2023-04-27 03:56:05,586 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/02f10fd10b534878a63b0658211e6a38 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/02f10fd10b534878a63b0658211e6a38 2023-04-27 03:56:05,589 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/02f10fd10b534878a63b0658211e6a38, entries=150, sequenceid=68, filesize=11.6 K 2023-04-27 03:56:05,589 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/c5aadee936d84e4fb683fc3a791f6a36 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/c5aadee936d84e4fb683fc3a791f6a36 2023-04-27 03:56:05,592 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/c5aadee936d84e4fb683fc3a791f6a36, entries=150, sequenceid=68, filesize=11.6 K 2023-04-27 03:56:05,593 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for 1c25ca7c0495563171ae7ccdf0350231 in 55ms, sequenceid=68, compaction requested=true 2023-04-27 03:56:05,593 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:05,593 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:05,593 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:05,593 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:05,593 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,593 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:05,593 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:05,595 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,595 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:05,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:05,596 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,596 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:05,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:05,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:05,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:05,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:05,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3993516b[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:05,597 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:05,598 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,599 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,599 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:05,599 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:05,599 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:05,599 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,599 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:05,600 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:05,600 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:05,600 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:05,600 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:05,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:05,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:05,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:05,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:05,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:05,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:05,603 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:05,603 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:05,603 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:05,603 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:05,603 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:05,605 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,605 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:05,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:05,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,605 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:05,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:05,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:05,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:05,605 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:05,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,607 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:05,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,609 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,610 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,610 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,610 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:05,609 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:05,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:05,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:05,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:05,610 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,610 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:05,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:05,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:05,610 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:05,610 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:05,610 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,610 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,611 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:05,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:05,711 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:05,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:05,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:05,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:05,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:05,714 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:05,714 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:05,714 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:05,714 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:05,714 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,714 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,716 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:05,716 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:05,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:05,716 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,716 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:05,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:05,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,717 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:05,717 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:05,717 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:05,717 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:05,717 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:05,717 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:05,717 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:05,717 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:05,718 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:05,718 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:05,718 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:05,718 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:05,718 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:05,718 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:05,718 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,718 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:05,718 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:05,719 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:05,720 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:05,720 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,720 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:05,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:05,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:05,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:05,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:05,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bc12e64[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:05,722 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:05,722 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,723 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:05,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:05,723 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,723 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:05,723 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:05,727 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:05,727 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:05,727 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,727 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:05,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:05,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:05,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:05,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:05,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:05,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:05,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:05,729 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:05,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:05,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:05,729 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:05,729 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:05,731 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,731 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:05,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:05,731 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:05,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,732 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:05,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:05,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:05,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:05,733 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:05,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,735 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,735 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:05,735 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:05,735 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:05,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:05,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:05,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:05,735 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:05,735 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,735 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,735 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,735 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,735 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:05,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:05,735 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:05,735 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:05,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:05,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:05,740 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:56:05,740 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:05,740 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:05,741 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:05,741 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:05,741 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:05,741 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:05,752 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427094541c0d4694af4831a491c746cbfb0 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427094541c0d4694af4831a491c746cbfb0 2023-04-27 03:56:05,758 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=79, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/098b019827f54f72bcf8954f6d5ed54c 2023-04-27 03:56:05,768 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/b9cd4b36ea0b42f6b0cf7130f052d398 2023-04-27 03:56:05,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:05,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567825768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:05,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48502 deadline: 1682567825769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567765761, max timestamp=1682567765771Num uniques 859; 2023-04-27 03:56:05,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:56:05,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567765761, max timestamp=1682567765771Num uniques 859; 2023-04-27 03:56:05,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:56:05,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567765761, max timestamp=1682567765771Num uniques 859; 2023-04-27 03:56:05,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:56:05,813 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/50b6bada450f4273a07b7516c6e05ce4 2023-04-27 03:56:05,817 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/098b019827f54f72bcf8954f6d5ed54c as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/098b019827f54f72bcf8954f6d5ed54c 2023-04-27 03:56:05,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682567765803, max timestamp=1682567765815Num uniques 1018; 2023-04-27 03:56:05,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682567765803, max timestamp=1682567765815Num uniques 1018; 2023-04-27 03:56:05,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-27 03:56:05,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-27 03:56:05,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682567765803, max timestamp=1682567765815Num uniques 1018; 2023-04-27 03:56:05,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-27 03:56:05,822 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/098b019827f54f72bcf8954f6d5ed54c, entries=250, sequenceid=79, filesize=38.7 K 2023-04-27 03:56:05,822 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/b9cd4b36ea0b42f6b0cf7130f052d398 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/b9cd4b36ea0b42f6b0cf7130f052d398 2023-04-27 03:56:05,826 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/b9cd4b36ea0b42f6b0cf7130f052d398, entries=100, sequenceid=79, filesize=9.3 K 2023-04-27 03:56:05,827 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/50b6bada450f4273a07b7516c6e05ce4 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/50b6bada450f4273a07b7516c6e05ce4 2023-04-27 03:56:05,829 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/50b6bada450f4273a07b7516c6e05ce4, entries=100, sequenceid=79, filesize=9.3 K 2023-04-27 03:56:05,830 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=67.09 KB/68700 for 1c25ca7c0495563171ae7ccdf0350231 in 90ms, sequenceid=79, compaction requested=true 2023-04-27 03:56:05,830 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:05,830 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-27 03:56:05,830 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-27 03:56:05,830 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-27 03:56:05,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:05,832 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=73.80 KB heapSize=200.16 KB 2023-04-27 03:56:05,832 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:05,833 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:05,833 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:05,833 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:05,833 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:05,833 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:05,836 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:05,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:05,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:05,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:05,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:05,840 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:05,840 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:05,840 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:05,840 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:05,840 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,840 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,845 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:05,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:05,845 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:05,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,845 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,845 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:05,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:05,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,846 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:05,846 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:05,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:05,846 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:05,846 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:05,846 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:05,846 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:05,846 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:05,846 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:05,847 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:05,847 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:05,847 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:05,847 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:05,847 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:05,847 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,847 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:05,847 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:05,848 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:05,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:05,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:05,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:05,849 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,850 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,850 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:05,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:05,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:05,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ec002c4[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:05,851 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:05,851 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,852 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,852 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:05,852 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:05,852 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:05,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:05,853 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427de470a56c3344e9f9178147dd3bcfa79 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427de470a56c3344e9f9178147dd3bcfa79 2023-04-27 03:56:05,853 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:05,853 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:05,853 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,854 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:05,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:05,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:05,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:05,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:05,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:05,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:05,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:05,856 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:05,856 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:05,856 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:05,856 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:05,856 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:05,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:05,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:05,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:05,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:05,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,860 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:05,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:05,861 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,863 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,863 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,863 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,863 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:05,863 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:05,863 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:05,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:05,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:05,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:05,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:05,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,864 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:05,864 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:05,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:05,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:05,878 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/20478bc9c95a4834938423a9e67e19b8 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/20478bc9c95a4834938423a9e67e19b8 2023-04-27 03:56:05,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:05,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48502 deadline: 1682567825875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:05,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567825875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:05,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567825879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:05,884 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/B of 1c25ca7c0495563171ae7ccdf0350231 into 20478bc9c95a4834938423a9e67e19b8(size=11.7 K), total size for store is 32.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:05,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567825879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,884 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:05,884 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/B, priority=13, startTime=1682567765449; duration=0sec 2023-04-27 03:56:05,885 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-27 03:56:05,885 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-04-27 03:56:05,886 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-27 03:56:05,886 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-27 03:56:05,886 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. because compaction request was cancelled 2023-04-27 03:56:05,886 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 03:56:05,888 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 03:56:05,888 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/C is initiating minor compaction (all files) 2023-04-27 03:56:05,888 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/C in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:05,889 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/dcbab4ee2e964367bce2defd9eb530dd, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/4897bb40512d499e90561b8c19fd87ef, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/c34472fdba6c428b8176053b2aedff7c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/c5aadee936d84e4fb683fc3a791f6a36, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/50b6bada450f4273a07b7516c6e05ce4] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=55.7 K 2023-04-27 03:56:05,889 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting dcbab4ee2e964367bce2defd9eb530dd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1682567762322 2023-04-27 03:56:05,889 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/fca51d0e7ade484b9d0cc1776d31dde9 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/fca51d0e7ade484b9d0cc1776d31dde9 2023-04-27 03:56:05,889 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 4897bb40512d499e90561b8c19fd87ef, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1682567762341 2023-04-27 03:56:05,890 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting c34472fdba6c428b8176053b2aedff7c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1682567764472 2023-04-27 03:56:05,891 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting c5aadee936d84e4fb683fc3a791f6a36, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682567764588 2023-04-27 03:56:05,891 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 50b6bada450f4273a07b7516c6e05ce4, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1682567765738 2023-04-27 03:56:05,899 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#C#compaction#533 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:05,906 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/A of 1c25ca7c0495563171ae7ccdf0350231 into fca51d0e7ade484b9d0cc1776d31dde9(size=25.2 K), total size for store is 89.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:05,906 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:05,906 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/A, priority=13, startTime=1682567765449; duration=0sec 2023-04-27 03:56:05,906 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-27 03:56:05,906 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 5 compacting, 0 eligible, 16 blocking 2023-04-27 03:56:05,906 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-27 03:56:05,906 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-27 03:56:05,906 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. because compaction request was cancelled 2023-04-27 03:56:05,906 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:56:05,907 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33417 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:56:05,907 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/B is initiating minor compaction (all files) 2023-04-27 03:56:05,907 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/B in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:05,907 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/20478bc9c95a4834938423a9e67e19b8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/02f10fd10b534878a63b0658211e6a38, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/b9cd4b36ea0b42f6b0cf7130f052d398] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=32.6 K 2023-04-27 03:56:05,907 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 20478bc9c95a4834938423a9e67e19b8, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1682567764472 2023-04-27 03:56:05,907 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 02f10fd10b534878a63b0658211e6a38, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682567764588 2023-04-27 03:56:05,908 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting b9cd4b36ea0b42f6b0cf7130f052d398, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1682567765738 2023-04-27 03:56:05,913 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#B#compaction#534 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:05,915 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/105a4e4d135f4f3eb0b350da9465f148 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/105a4e4d135f4f3eb0b350da9465f148 2023-04-27 03:56:05,919 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/C of 1c25ca7c0495563171ae7ccdf0350231 into 105a4e4d135f4f3eb0b350da9465f148(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:05,919 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:05,919 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/C, priority=11, startTime=1682567765830; duration=0sec 2023-04-27 03:56:05,919 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:05,964 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:05,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:05,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:05,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:05,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:05,968 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:05,968 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:05,968 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:05,968 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:05,969 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,969 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,970 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:05,970 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:05,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:05,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,971 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,971 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:05,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:05,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:05,972 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:05,972 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:05,972 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:05,972 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:05,972 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:05,972 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:05,972 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:05,972 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:05,972 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:05,972 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:05,972 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:05,972 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:05,973 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:05,973 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,973 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:05,973 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:05,974 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,974 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:05,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:05,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:05,975 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,975 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:05,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:05,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:05,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:05,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e3a399a[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:05,976 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:05,976 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,979 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:05,979 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,979 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:05,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:05,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:05,980 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:05,981 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:05,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:05,981 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:05,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:05,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:05,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:05,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:05,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:05,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:05,983 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:05,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:05,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:05,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:05,984 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:05,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567825982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,984 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:05,985 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,985 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:05,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:05,985 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:05,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:05,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:05,986 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:05,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:05,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567825985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:05,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567825985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:05,987 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:05,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:05,989 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:05,989 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:05,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:05,990 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:05,989 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,990 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:05,990 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:05,990 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,990 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:05,990 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:05,989 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:05,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:05,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:05,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:05,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:05,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:05,990 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:05,990 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:06,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:06,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48502 deadline: 1682567826080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,103 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:06,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:06,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:06,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:06,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:06,107 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:06,107 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:06,107 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:06,107 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:06,108 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,108 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:06,110 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:06,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:06,111 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,111 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:06,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:06,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:06,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:06,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:06,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:06,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:06,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:06,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:06,121 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:06,121 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:06,121 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:06,121 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:06,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:06,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:06,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:06,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:06,123 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,123 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:06,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:06,123 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,123 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:06,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:06,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:06,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:06,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:06,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d812b3f[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:06,125 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:06,125 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,129 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,129 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:06,129 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:06,129 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:06,129 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,129 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:06,130 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,130 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:06,130 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:06,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:06,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:06,130 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:06,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:06,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:06,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:06,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:06,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:06,133 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:06,133 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:06,133 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:06,133 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:06,133 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:06,135 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,135 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:06,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:06,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:06,136 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,136 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:06,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:06,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:06,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:06,137 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:06,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,139 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,139 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,140 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:06,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:06,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:06,140 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:06,140 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:06,140 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:06,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:06,140 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:06,140 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,140 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:06,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:06,140 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,140 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,140 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:06,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:06,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:06,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567826186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:06,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567826189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:06,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567826189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,240 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:06,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:06,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:06,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:06,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:06,244 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:06,244 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:06,244 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:06,244 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:06,244 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,244 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,245 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,245 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:06,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:06,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:06,246 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,246 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:06,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:06,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:06,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:06,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:06,247 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:06,247 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:06,247 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:06,247 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:06,247 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:06,247 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:06,247 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:06,248 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:06,248 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:06,248 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:06,248 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:06,248 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,248 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:06,248 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:06,258 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,258 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:06,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:06,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:06,258 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,259 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:06,259 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=123, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/3bd85fdf4a6e403688579f1915a6fa86 2023-04-27 03:56:06,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:06,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:06,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:06,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f4180e5[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:06,261 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:06,261 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,262 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,262 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,262 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:06,262 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:06,262 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:06,262 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:06,264 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:06,264 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:06,264 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,264 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:06,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:06,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:06,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:06,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:06,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:06,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:06,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:06,267 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:06,267 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:06,267 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:06,267 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:06,267 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:06,269 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,269 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:06,269 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:06,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:06,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:06,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:06,269 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:06,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:06,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,271 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:06,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,274 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:06,274 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,274 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,274 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:06,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:06,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:06,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:06,274 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:06,274 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:06,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:06,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:06,274 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:06,274 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,275 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,275 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,275 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:06,275 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:06,326 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/48a5c1d239384ba1a55278f2707911dc as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/48a5c1d239384ba1a55278f2707911dc 2023-04-27 03:56:06,330 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/B of 1c25ca7c0495563171ae7ccdf0350231 into 48a5c1d239384ba1a55278f2707911dc(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:06,330 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:06,330 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/B, priority=13, startTime=1682567765830; duration=0sec 2023-04-27 03:56:06,330 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:06,375 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:06,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:06,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:06,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:06,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:06,378 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:06,378 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:06,378 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:06,378 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:06,378 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,378 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,379 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:06,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:06,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:06,380 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,380 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:06,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:06,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:06,381 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:06,381 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:06,381 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:06,381 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:06,381 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:06,381 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:06,381 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:06,381 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:06,381 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:06,381 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:06,381 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:06,381 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:06,381 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:06,381 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,382 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:06,382 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:06,383 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:06,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:06,383 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:06,383 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,384 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:06,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:06,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:06,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:06,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51096deb[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:06,385 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:06,385 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,386 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,386 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,386 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:06,386 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:06,386 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:06,386 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:06,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:06,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48502 deadline: 1682567826385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,388 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:06,388 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,388 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:06,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:06,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:06,389 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:06,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:06,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:06,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:06,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:06,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:06,391 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:06,391 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:06,391 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:06,391 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:06,391 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:06,392 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,392 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:06,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,392 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:06,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:06,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:06,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:06,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:06,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,393 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:06,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,394 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:06,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,396 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:06,396 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:06,396 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:06,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:06,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:06,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:06,396 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:06,396 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,396 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,396 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,396 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:06,396 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:06,396 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,396 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,396 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:06,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:06,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:06,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567826489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:06,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567826492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:06,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567826494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:06,496 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:06,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48474 deadline: 1682567826496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,498 DEBUG [Thread-4053] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., hostname=jenkins-hbase4.apache.org,40885,1682567618305, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 03:56:06,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:06,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:06,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:06,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:06,501 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:06,501 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:06,501 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:06,501 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:06,501 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,501 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,502 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:06,503 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:06,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:06,503 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,503 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:06,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:06,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:06,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:06,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:06,504 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:06,504 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:06,504 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:06,504 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:06,504 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:06,504 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:06,504 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:06,504 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:06,504 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:06,504 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:06,504 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:06,504 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,505 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:06,505 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:06,507 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:06,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:06,508 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:06,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:06,508 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,508 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:06,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:06,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:06,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e3078b9[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:06,509 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:06,509 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,510 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,510 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,510 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:06,510 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:06,510 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:06,510 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:06,512 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:06,512 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,512 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:06,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:06,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:06,512 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:06,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:06,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:06,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:06,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:06,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:06,514 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:06,514 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:06,514 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:06,514 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:06,514 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:06,527 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,527 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,527 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:06,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:06,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:06,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,528 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:06,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:06,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:06,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:06,529 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:06,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,533 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:06,533 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:06,533 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,533 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,533 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,533 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,533 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,533 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,533 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:06,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:06,533 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:06,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:06,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:06,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:06,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:06,533 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:06,533 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:06,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:06,633 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:06,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:06,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:06,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:06,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:06,637 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:06,637 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:06,637 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:06,637 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:06,637 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,637 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,638 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:06,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:06,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:06,638 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,639 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,639 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:06,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:06,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,640 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:06,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:06,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:06,640 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:06,640 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:06,640 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:06,640 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:06,641 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:06,641 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:06,641 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:06,641 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:06,641 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:06,641 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:06,641 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:06,641 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,641 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:06,641 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:06,642 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,642 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:06,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:06,643 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,643 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:06,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:06,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:06,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:06,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:06,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a5c6093[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:06,644 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:06,644 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,645 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,645 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:06,645 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:06,646 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:06,646 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,646 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:06,650 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:06,650 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:06,650 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,650 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:06,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:06,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:06,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:06,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:06,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:06,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:06,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:06,653 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:06,653 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:06,653 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:06,653 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:06,653 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:06,654 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,654 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:06,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,655 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:06,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:06,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:06,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:06,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:06,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,655 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:06,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:06,657 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,659 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,659 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,659 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:06,659 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:06,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:06,659 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:06,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:06,659 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:06,659 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:06,659 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:06,659 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,659 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,659 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,659 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:06,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:06,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:06,682 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/28a6235d9a9c4bb7a102de16cf70bc10 2023-04-27 03:56:06,698 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/7900f9725daf442699d4a43379d5d503 2023-04-27 03:56:06,702 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/3bd85fdf4a6e403688579f1915a6fa86 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/3bd85fdf4a6e403688579f1915a6fa86 2023-04-27 03:56:06,705 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/3bd85fdf4a6e403688579f1915a6fa86, entries=200, sequenceid=123, filesize=31.9 K 2023-04-27 03:56:06,706 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/28a6235d9a9c4bb7a102de16cf70bc10 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/28a6235d9a9c4bb7a102de16cf70bc10 2023-04-27 03:56:06,709 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/28a6235d9a9c4bb7a102de16cf70bc10, entries=150, sequenceid=123, filesize=11.6 K 2023-04-27 03:56:06,709 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/7900f9725daf442699d4a43379d5d503 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/7900f9725daf442699d4a43379d5d503 2023-04-27 03:56:06,712 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/7900f9725daf442699d4a43379d5d503, entries=150, sequenceid=123, filesize=11.6 K 2023-04-27 03:56:06,713 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~200.11 KB/204912, currentSize=114.05 KB/116790 for 1c25ca7c0495563171ae7ccdf0350231 in 881ms, sequenceid=123, compaction requested=true 2023-04-27 03:56:06,713 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:06,713 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:06,713 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:06,713 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 03:56:06,713 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-27 03:56:06,713 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-27 03:56:06,714 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-27 03:56:06,714 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-27 03:56:06,714 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. because compaction request was cancelled 2023-04-27 03:56:06,714 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-27 03:56:06,714 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 123907 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 03:56:06,714 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/A is initiating minor compaction (all files) 2023-04-27 03:56:06,714 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/A in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:06,714 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/fca51d0e7ade484b9d0cc1776d31dde9, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/7606d53b0e1a45c4a0dd8512512bb7cf, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/098b019827f54f72bcf8954f6d5ed54c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/3bd85fdf4a6e403688579f1915a6fa86] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=121.0 K 2023-04-27 03:56:06,714 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-27 03:56:06,715 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-27 03:56:06,715 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. because compaction request was cancelled 2023-04-27 03:56:06,715 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting fca51d0e7ade484b9d0cc1776d31dde9, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1682567764472 2023-04-27 03:56:06,715 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 7606d53b0e1a45c4a0dd8512512bb7cf, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682567764588 2023-04-27 03:56:06,716 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 098b019827f54f72bcf8954f6d5ed54c, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1682567765723 2023-04-27 03:56:06,716 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 3bd85fdf4a6e403688579f1915a6fa86, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1682567765815 2023-04-27 03:56:06,725 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#A#compaction#537 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:06,760 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:06,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:06,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:06,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:06,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:06,762 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:06,762 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:06,762 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:06,763 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:06,763 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,763 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,764 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:06,764 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:06,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:06,764 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,764 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:06,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:06,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,765 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:06,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:06,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:06,766 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:06,766 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:06,766 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:06,766 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:06,766 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:06,766 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:06,767 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:06,767 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-27 03:56:06,767 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:06,767 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:06,767 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:06,767 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:06,767 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:06,767 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:06,785 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042778050e411c14453aa30f3a5cc36cbe33 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042778050e411c14453aa30f3a5cc36cbe33 2023-04-27 03:56:06,798 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=147, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/26ebed07f6934b2899d1e10234ced1ed 2023-04-27 03:56:06,807 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/310071fdb6c4440b83b1fcdcc6d9155a 2023-04-27 03:56:06,816 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/0aad3a81e7e649fa82ad7a545a081c6d 2023-04-27 03:56:06,819 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/26ebed07f6934b2899d1e10234ced1ed as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/26ebed07f6934b2899d1e10234ced1ed 2023-04-27 03:56:06,823 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/26ebed07f6934b2899d1e10234ced1ed, entries=150, sequenceid=147, filesize=25.3 K 2023-04-27 03:56:06,823 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/310071fdb6c4440b83b1fcdcc6d9155a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/310071fdb6c4440b83b1fcdcc6d9155a 2023-04-27 03:56:06,826 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/310071fdb6c4440b83b1fcdcc6d9155a, entries=150, sequenceid=147, filesize=11.8 K 2023-04-27 03:56:06,827 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/0aad3a81e7e649fa82ad7a545a081c6d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/0aad3a81e7e649fa82ad7a545a081c6d 2023-04-27 03:56:06,837 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/0aad3a81e7e649fa82ad7a545a081c6d, entries=150, sequenceid=147, filesize=11.8 K 2023-04-27 03:56:06,838 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 1c25ca7c0495563171ae7ccdf0350231 in 71ms, sequenceid=147, compaction requested=true 2023-04-27 03:56:06,838 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:06,838 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:06,838 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:06,838 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:06,838 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,838 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:06,838 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:06,840 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,840 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:06,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:06,841 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:06,841 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:06,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:06,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:06,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:06,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69d98b95[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:06,842 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:06,843 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,844 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,844 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:06,844 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,844 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:06,844 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:06,844 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:06,845 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:06,845 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,845 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:06,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:06,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:06,845 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:06,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:06,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:06,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:06,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:06,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:06,848 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:06,848 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:06,848 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:06,848 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:06,848 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:06,849 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:06,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:06,849 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,850 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:06,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:06,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:06,850 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:06,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:06,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,851 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:06,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:06,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:06,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:06,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:06,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:06,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:06,859 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:06,859 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:06,859 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:06,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,859 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,860 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,860 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:06,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:06,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:06,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:06,937 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:56:06,938 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:06,938 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:06,938 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:06,938 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:06,938 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:06,938 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:06,959 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:06,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:06,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:06,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:06,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:06,962 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:06,962 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:06,962 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:06,962 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:06,963 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,963 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,964 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:06,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:06,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:06,964 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,964 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,965 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:06,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:06,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:06,965 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:06,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:06,965 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:06,965 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:06,965 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:06,966 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:06,966 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:06,966 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:06,966 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:06,966 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:06,966 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:06,966 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:06,966 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:06,966 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,966 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:06,966 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:06,968 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:06,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:06,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,968 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:06,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:06,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:06,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:06,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:06,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73a5c570[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:06,970 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:06,970 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,971 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:06,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:06,971 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:06,971 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,971 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:06,973 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:06,973 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:06,973 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,973 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:06,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:06,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:06,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:06,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:06,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:06,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:06,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:06,975 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:06,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:06,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:06,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:06,975 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:06,976 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,976 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:06,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:06,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,977 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:06,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:06,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:06,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:06,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:06,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:06,979 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:06,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,981 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,981 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,981 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:06,981 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:06,981 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:06,981 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:06,981 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:06,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:06,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:06,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:06,981 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:06,981 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:06,981 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:06,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:06,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:06,981 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:06,981 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:07,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567766972, max timestamp=1682567766998Num uniques 859; 2023-04-27 03:56:07,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567766972, max timestamp=1682567766998Num uniques 859; 2023-04-27 03:56:07,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567766972, max timestamp=1682567766998Num uniques 859; 2023-04-27 03:56:07,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:56:07,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:56:07,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:56:07,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:07,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567827015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:07,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567827016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:07,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48502 deadline: 1682567827016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:07,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567827018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,081 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:07,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:07,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:07,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:07,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:07,085 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:07,085 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:07,085 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:07,085 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:07,086 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,086 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,087 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:07,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:07,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:07,088 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,088 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,088 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:07,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:07,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,088 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:07,089 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:07,089 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:07,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:07,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:07,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:07,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:07,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:07,089 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:07,089 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:07,089 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:07,089 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:07,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:07,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:07,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:07,089 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:07,092 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,092 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:07,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:07,092 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,092 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:07,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:07,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:07,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:07,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:07,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e4d9e0d[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:07,094 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:07,095 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,096 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,096 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:07,096 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:07,096 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:07,096 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,096 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:07,098 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,098 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:07,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:07,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:07,098 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:07,098 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:07,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:07,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:07,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:07,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:07,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:07,104 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:07,104 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:07,105 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:07,105 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:07,105 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:07,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:07,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:07,111 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,111 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:07,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:07,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:07,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:07,112 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:07,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:07,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,118 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:07,118 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:07,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:07,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:07,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:07,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:07,120 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,120 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,120 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,120 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,120 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,120 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,121 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,121 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:07,121 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:07,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:07,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:07,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567827120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,121 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:07,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567827120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,121 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:07,121 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:07,121 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:07,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48502 deadline: 1682567827120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:07,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567827122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,145 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/26341f746d034850862c6645b571232b as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/26341f746d034850862c6645b571232b 2023-04-27 03:56:07,149 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/A of 1c25ca7c0495563171ae7ccdf0350231 into 26341f746d034850862c6645b571232b(size=25.4 K), total size for store is 50.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:07,149 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:07,149 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/A, priority=12, startTime=1682567766713; duration=0sec 2023-04-27 03:56:07,149 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:07,221 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:07,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:07,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:07,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:07,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:07,225 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:07,225 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:07,225 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:07,225 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:07,225 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,226 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,227 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:07,227 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:07,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:07,227 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,227 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:07,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:07,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:07,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:07,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:07,228 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:07,228 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:07,228 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:07,228 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:07,228 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:07,228 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:07,228 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:07,229 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:07,229 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:07,229 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:07,229 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:07,229 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,229 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:07,229 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:07,230 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,230 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:07,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:07,230 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,230 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:07,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:07,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:07,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:07,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:07,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27dc23af[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:07,232 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:07,232 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,234 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,234 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,234 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:07,234 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:07,234 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:07,235 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:07,236 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:07,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:07,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:07,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:07,236 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:07,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:07,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:07,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:07,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:07,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:07,238 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:07,239 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:07,239 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:07,239 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:07,239 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:07,240 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,240 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,240 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:07,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:07,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:07,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:07,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:07,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:07,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:07,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,242 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:07,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,244 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,244 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:07,244 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,244 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,244 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,244 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:07,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:07,244 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:07,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:07,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:07,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:07,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:07,244 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,244 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,245 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,245 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:07,245 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:07,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:07,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:07,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:07,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48502 deadline: 1682567827322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:07,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567827323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:07,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567827323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:07,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567827325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,345 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:07,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:07,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:07,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:07,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:07,348 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:07,348 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:07,348 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:07,348 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:07,349 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,349 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,349 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304276c335d1460c54ba3b7305bc0d612a97f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304276c335d1460c54ba3b7305bc0d612a97f 2023-04-27 03:56:07,355 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,355 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:07,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:07,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:07,355 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,355 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:07,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:07,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:07,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:07,357 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:07,357 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:07,357 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:07,357 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:07,357 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:07,357 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:07,357 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:07,357 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:07,357 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:07,357 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:07,357 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:07,357 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:07,357 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,357 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:07,357 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:07,359 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:07,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:07,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:07,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:07,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:07,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:07,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:07,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27c81e2[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:07,361 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:07,361 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,362 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,362 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:07,362 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:07,362 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:07,362 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,362 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:07,363 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,363 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:07,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:07,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:07,364 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:07,364 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:07,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:07,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:07,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:07,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:07,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:07,366 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:07,366 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:07,366 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:07,366 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:07,366 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:07,368 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,368 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,369 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:07,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:07,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:07,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:07,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:07,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,370 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:07,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:07,371 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:07,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,373 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:07,373 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,373 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,373 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,373 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,373 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,373 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,373 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,373 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:07,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:07,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:07,373 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:07,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:07,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:07,373 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:07,373 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:07,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:07,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:07,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:07,474 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:07,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:07,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:07,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:07,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:07,477 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:07,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:07,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:07,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:07,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,477 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,478 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:07,478 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:07,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:07,479 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,479 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:07,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:07,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:07,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:07,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:07,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:07,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:07,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:07,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:07,480 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:07,480 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:07,480 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:07,481 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:07,481 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:07,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:07,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:07,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:07,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:07,482 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:07,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:07,482 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:07,483 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,483 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:07,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:07,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:07,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:07,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f2f4a98[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:07,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:07,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,487 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,487 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:07,487 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:07,487 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,487 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:07,487 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:07,488 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,488 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:07,488 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:07,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:07,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:07,488 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:07,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:07,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:07,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:07,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:07,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:07,491 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:07,491 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:07,491 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:07,491 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:07,491 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:07,493 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,493 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,493 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:07,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:07,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:07,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:07,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:07,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:07,494 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:07,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,495 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:07,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,502 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,502 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,502 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,502 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,502 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:07,502 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:07,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:07,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:07,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:07,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:07,503 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:07,502 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,503 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,503 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,503 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:07,503 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:07,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:07,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:07,603 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:07,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:07,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:07,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:07,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:07,606 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:07,606 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:07,606 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:07,606 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:07,607 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,607 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,608 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:07,608 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:07,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:07,608 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,608 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:07,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:07,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:07,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:07,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:07,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:07,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:07,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:07,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:07,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:07,610 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:07,610 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:07,610 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:07,610 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:07,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:07,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:07,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,611 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:07,611 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:07,612 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,612 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:07,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:07,612 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,612 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:07,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:07,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:07,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:07,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:07,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d101a86[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:07,614 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:07,614 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,615 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,615 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:07,615 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:07,615 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:07,615 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,615 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:07,617 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:07,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:07,617 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:07,617 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:07,617 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:07,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:07,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:07,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:07,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:07,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:07,620 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:07,620 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:07,620 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:07,620 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:07,620 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:07,621 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:07,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:07,621 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,621 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:07,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:07,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:07,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:07,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,622 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:07,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,623 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:07,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,625 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,625 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:07,625 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,625 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:07,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:07,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:07,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:07,625 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:07,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:07,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:07,625 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,626 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,626 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:07,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:07,626 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:07,626 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:07,626 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,626 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:07,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48502 deadline: 1682567827625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:07,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567827626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:07,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567827627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:07,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567827629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,721 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-27 03:56:07,726 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:07,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:07,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:07,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:07,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:07,740 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:07,740 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:07,740 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:07,740 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:07,741 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,741 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,742 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:07,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:07,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:07,742 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,743 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,743 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:07,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:07,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,744 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:07,752 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:07,752 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:07,752 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:07,752 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:07,752 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:07,753 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:07,753 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:07,753 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:07,753 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:07,753 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:07,753 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:07,753 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:07,753 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:07,753 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,753 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:07,753 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:07,756 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=157, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/6b58444e2dbb4f56b8c8a5a1819272ba 2023-04-27 03:56:07,756 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,756 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:07,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:07,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:07,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:07,757 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,757 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:07,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:07,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:07,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b178bd0[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:07,758 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:07,758 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,759 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,759 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:07,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:07,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:07,760 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,760 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:07,761 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:07,761 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:07,761 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:07,761 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:07,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:07,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:07,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:07,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:07,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:07,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:07,765 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:07,765 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:07,765 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:07,765 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:07,765 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:07,767 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,767 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:07,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,767 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:07,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:07,767 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:07,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,769 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:07,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,772 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:07,772 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:07,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:07,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:07,772 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,773 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:07,773 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:07,773 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:07,773 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:07,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:07,773 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:07,773 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/a7e7f0825c6940399537fe650f9024b8 2023-04-27 03:56:07,773 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,773 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,773 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,773 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:07,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:07,873 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:07,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:07,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:07,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:07,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:07,876 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:07,876 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:07,877 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:07,877 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:07,877 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,877 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,878 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:07,878 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:07,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:07,879 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,879 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:07,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:07,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:07,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:07,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:07,880 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:07,880 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:07,880 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:07,880 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:07,880 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:07,880 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:07,880 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:07,880 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:07,880 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:07,880 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:07,880 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:07,880 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,881 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:07,881 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:07,882 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,882 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:07,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:07,883 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,883 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:07,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:07,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:07,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:07,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:07,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d827ac1[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:07,884 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:07,884 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,885 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,886 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:07,886 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:07,886 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:07,886 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,886 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:07,890 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:07,891 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:07,891 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:07,891 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:07,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:07,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:07,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:07,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:07,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:07,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:07,893 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:07,893 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:07,893 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:07,894 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:07,894 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:07,895 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,895 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:07,895 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:07,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:07,895 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:07,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:07,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:07,896 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:07,898 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:07,898 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,898 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:07,899 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:07,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:07,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:07,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:07,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:07,899 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:07,899 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:07,899 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:07,899 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,899 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:07,899 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:07,899 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:07,899 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:07,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:07,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:07,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:07,999 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:08,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:08,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:08,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:08,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:08,003 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:08,003 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:08,003 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:08,003 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:08,003 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,003 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,012 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:08,012 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:08,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:08,013 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,013 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:08,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:08,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:08,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:08,014 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:08,014 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:08,014 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:08,014 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:08,014 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:08,014 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:08,014 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:08,014 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:08,014 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:08,014 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:08,015 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:08,015 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,015 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:08,015 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:08,016 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,016 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:08,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:08,016 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,016 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:08,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:08,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:08,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:08,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:08,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@585c8463[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:08,018 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:08,018 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,020 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,020 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:08,020 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,020 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:08,020 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:08,020 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:08,021 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:08,021 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,021 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:08,021 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:08,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:08,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:08,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:08,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:08,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:08,024 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:08,024 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:08,024 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:08,024 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:08,024 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:08,027 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,027 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:08,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,027 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:08,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:08,028 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,028 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:08,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,030 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,030 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,031 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,031 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:08,031 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:08,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:08,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:08,031 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:08,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:08,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:08,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:08,031 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,031 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,031 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,031 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,031 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:08,031 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:08,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:08,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:08,131 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:08,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:08,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:08,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:08,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:08,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:08,134 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:08,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48502 deadline: 1682567828131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,135 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:08,135 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:08,135 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:08,135 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:08,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567828132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,135 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,135 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:08,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567828132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,136 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:08,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567828135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,136 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,136 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:08,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:08,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:08,137 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,137 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:08,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:08,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:08,137 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:08,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:08,137 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:08,137 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:08,138 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:08,138 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:08,138 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:08,138 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:08,138 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:08,138 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:08,138 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:08,138 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:08,138 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:08,138 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,138 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:08,138 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:08,140 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,140 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:08,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:08,140 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,140 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:08,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:08,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:08,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:08,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:08,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54571c4[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:08,141 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:08,141 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,142 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,143 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:08,143 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:08,143 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:08,143 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,143 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:08,144 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,144 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:08,145 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:08,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,145 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:08,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:08,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:08,147 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:08,147 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:08,147 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:08,147 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:08,147 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:08,148 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,148 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:08,148 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:08,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:08,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:08,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:08,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:08,149 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:08,150 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:08,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,152 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,152 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,152 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,152 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,152 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:08,152 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:08,152 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,152 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,152 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:08,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:08,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:08,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:08,152 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:08,153 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:08,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:08,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:08,153 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:08,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:08,185 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/8e23f2aa8d3d4153bbd3081168f72755 2023-04-27 03:56:08,189 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/6b58444e2dbb4f56b8c8a5a1819272ba as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/6b58444e2dbb4f56b8c8a5a1819272ba 2023-04-27 03:56:08,192 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/6b58444e2dbb4f56b8c8a5a1819272ba, entries=300, sequenceid=157, filesize=45.8 K 2023-04-27 03:56:08,193 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/a7e7f0825c6940399537fe650f9024b8 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/a7e7f0825c6940399537fe650f9024b8 2023-04-27 03:56:08,195 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/a7e7f0825c6940399537fe650f9024b8, entries=100, sequenceid=157, filesize=9.4 K 2023-04-27 03:56:08,196 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/8e23f2aa8d3d4153bbd3081168f72755 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/8e23f2aa8d3d4153bbd3081168f72755 2023-04-27 03:56:08,199 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/8e23f2aa8d3d4153bbd3081168f72755, entries=100, sequenceid=157, filesize=9.4 K 2023-04-27 03:56:08,199 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1c25ca7c0495563171ae7ccdf0350231 in 1262ms, sequenceid=157, compaction requested=true 2023-04-27 03:56:08,199 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:08,199 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:08,199 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:56:08,200 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 03:56:08,200 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:08,200 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:08,200 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 98748 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:56:08,200 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/A is initiating minor compaction (all files) 2023-04-27 03:56:08,201 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/A in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:08,201 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/26341f746d034850862c6645b571232b, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/26ebed07f6934b2899d1e10234ced1ed, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/6b58444e2dbb4f56b8c8a5a1819272ba] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=96.4 K 2023-04-27 03:56:08,201 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45656 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 03:56:08,201 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/B is initiating minor compaction (all files) 2023-04-27 03:56:08,201 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/B in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:08,201 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/48a5c1d239384ba1a55278f2707911dc, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/28a6235d9a9c4bb7a102de16cf70bc10, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/310071fdb6c4440b83b1fcdcc6d9155a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/a7e7f0825c6940399537fe650f9024b8] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=44.6 K 2023-04-27 03:56:08,201 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 26341f746d034850862c6645b571232b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1682567765815 2023-04-27 03:56:08,201 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 48a5c1d239384ba1a55278f2707911dc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1682567764588 2023-04-27 03:56:08,202 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 26ebed07f6934b2899d1e10234ced1ed, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1682567765873 2023-04-27 03:56:08,202 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 28a6235d9a9c4bb7a102de16cf70bc10, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1682567765815 2023-04-27 03:56:08,202 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 6b58444e2dbb4f56b8c8a5a1819272ba, keycount=300, bloomtype=ROW, size=45.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682567766919 2023-04-27 03:56:08,202 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 310071fdb6c4440b83b1fcdcc6d9155a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1682567765873 2023-04-27 03:56:08,202 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting a7e7f0825c6940399537fe650f9024b8, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682567766932 2023-04-27 03:56:08,209 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#B#compaction#545 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:08,213 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#A#compaction#544 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:08,253 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:08,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:08,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:08,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:08,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:08,256 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:08,256 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:08,256 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:08,256 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:08,256 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,256 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,258 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,258 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:08,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:08,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:08,258 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,258 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:08,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:08,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:08,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:08,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:08,259 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:08,259 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:08,260 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:08,260 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:08,260 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:08,260 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:08,260 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:08,260 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-27 03:56:08,260 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:08,260 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:08,260 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:08,260 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:08,260 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:08,261 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:08,276 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427ea9878a2842641e6ba33b69354170b46 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427ea9878a2842641e6ba33b69354170b46 2023-04-27 03:56:08,281 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=194, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/dd922210d84c45b3a976705dc732d605 2023-04-27 03:56:08,625 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/45a54d3a10074cf9a0fe196986505ce6 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/45a54d3a10074cf9a0fe196986505ce6 2023-04-27 03:56:08,629 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/B of 1c25ca7c0495563171ae7ccdf0350231 into 45a54d3a10074cf9a0fe196986505ce6(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:08,629 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:08,629 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/B, priority=12, startTime=1682567768199; duration=0sec 2023-04-27 03:56:08,629 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:08,629 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-27 03:56:08,631 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45622 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-27 03:56:08,632 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/C is initiating minor compaction (all files) 2023-04-27 03:56:08,632 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/C in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:08,632 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/105a4e4d135f4f3eb0b350da9465f148, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/7900f9725daf442699d4a43379d5d503, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/0aad3a81e7e649fa82ad7a545a081c6d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/8e23f2aa8d3d4153bbd3081168f72755] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=44.6 K 2023-04-27 03:56:08,632 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 105a4e4d135f4f3eb0b350da9465f148, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1682567764588 2023-04-27 03:56:08,633 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 7900f9725daf442699d4a43379d5d503, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1682567765815 2023-04-27 03:56:08,633 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 0aad3a81e7e649fa82ad7a545a081c6d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1682567765873 2023-04-27 03:56:08,634 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 8e23f2aa8d3d4153bbd3081168f72755, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682567766932 2023-04-27 03:56:08,638 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/9fc988a166f544ae9c2dcf6d5aa16381 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/9fc988a166f544ae9c2dcf6d5aa16381 2023-04-27 03:56:08,657 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#C#compaction#548 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:08,659 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/A of 1c25ca7c0495563171ae7ccdf0350231 into 9fc988a166f544ae9c2dcf6d5aa16381(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:08,659 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:08,659 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/A, priority=13, startTime=1682567768199; duration=0sec 2023-04-27 03:56:08,660 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:08,666 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/a566b15837784089bfd58229b241268e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a566b15837784089bfd58229b241268e 2023-04-27 03:56:08,670 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/C of 1c25ca7c0495563171ae7ccdf0350231 into a566b15837784089bfd58229b241268e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:08,670 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:08,671 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/C, priority=12, startTime=1682567768200; duration=0sec 2023-04-27 03:56:08,671 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:08,701 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/216f8fec746d46beb3ff7e482ce64b80 2023-04-27 03:56:08,711 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/a7d3b09ec7ad4e998d9939e004b7f1ef 2023-04-27 03:56:08,714 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/dd922210d84c45b3a976705dc732d605 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/dd922210d84c45b3a976705dc732d605 2023-04-27 03:56:08,718 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/dd922210d84c45b3a976705dc732d605, entries=150, sequenceid=194, filesize=25.3 K 2023-04-27 03:56:08,718 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/216f8fec746d46beb3ff7e482ce64b80 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/216f8fec746d46beb3ff7e482ce64b80 2023-04-27 03:56:08,722 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/216f8fec746d46beb3ff7e482ce64b80, entries=150, sequenceid=194, filesize=11.8 K 2023-04-27 03:56:08,723 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/a7d3b09ec7ad4e998d9939e004b7f1ef as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a7d3b09ec7ad4e998d9939e004b7f1ef 2023-04-27 03:56:08,726 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a7d3b09ec7ad4e998d9939e004b7f1ef, entries=150, sequenceid=194, filesize=11.8 K 2023-04-27 03:56:08,727 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=0 B/0 for 1c25ca7c0495563171ae7ccdf0350231 in 467ms, sequenceid=194, compaction requested=false 2023-04-27 03:56:08,727 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:08,727 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:08,727 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:08,727 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:08,727 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,727 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:08,727 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:08,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,729 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:08,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:08,730 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,730 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:08,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:08,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:08,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:08,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:08,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@222b33b7[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:08,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:08,731 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,733 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,733 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:08,733 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:08,733 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:08,733 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,733 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:08,734 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,734 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:08,734 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:08,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:08,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:08,734 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:08,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:08,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:08,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:08,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:08,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:08,737 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:08,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:08,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:08,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:08,737 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:08,739 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,739 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:08,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:08,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,739 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:08,739 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:08,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:08,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:08,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:08,740 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:08,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,743 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,744 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,744 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:08,744 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:08,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:08,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:08,744 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,744 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,744 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:08,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:08,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:08,744 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:08,744 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:08,744 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:08,744 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,744 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:08,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:08,844 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:08,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:08,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:08,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:08,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:08,848 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:08,848 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:08,848 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:08,848 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:08,848 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,848 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,849 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:08,849 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:08,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:08,850 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,850 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:08,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:08,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:08,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:08,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:08,851 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:08,851 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:08,851 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:08,851 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:08,851 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:08,851 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:08,851 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:08,851 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:08,852 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:08,852 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:08,852 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:08,852 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,852 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:08,852 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:08,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,856 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:08,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:08,857 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,857 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:08,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:08,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:08,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:08,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:08,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c111141[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:08,858 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:08,858 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,860 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,860 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:08,860 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:08,861 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:08,861 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,861 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:08,862 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:08,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:08,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:08,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:08,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:08,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:08,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:08,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:08,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:08,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:08,864 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:08,865 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:08,865 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:08,865 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:08,865 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:08,866 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,866 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,866 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:08,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:08,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:08,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:08,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:08,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:08,867 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:08,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,868 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:08,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:08,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:08,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:08,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:08,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:08,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:08,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:08,870 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,871 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,871 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,870 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:08,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:08,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:08,871 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:08,871 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:08,971 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:08,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:08,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:08,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:08,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:08,975 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:08,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:08,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:08,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:08,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,975 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,977 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:08,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:08,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:08,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:08,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:08,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:08,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:08,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:08,978 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:08,978 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:08,978 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:08,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:08,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:08,979 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:08,979 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:08,979 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:08,979 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:08,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:08,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:08,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:08,979 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:08,980 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,981 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:08,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:08,981 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,981 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:08,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:08,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:08,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:08,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:08,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7052fb3c[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:08,983 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:08,983 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,984 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,984 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:08,984 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:08,984 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:08,985 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,985 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:08,986 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,986 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:08,986 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:08,986 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:08,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:08,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:08,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:08,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:08,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:08,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:08,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:08,989 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:08,989 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:08,989 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:08,989 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:08,989 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:08,991 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,991 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:08,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:08,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,991 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:08,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:08,992 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:08,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:08,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:08,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,993 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:08,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:08,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,995 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:08,995 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:08,995 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:08,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:08,996 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:08,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:08,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:08,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:08,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:08,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:08,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:08,996 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:08,996 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:08,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:08,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:08,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:09,096 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:09,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:09,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:09,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:09,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:09,100 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:09,100 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:09,100 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:09,100 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:09,100 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,100 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,101 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:09,101 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:09,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:09,102 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,102 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:09,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:09,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:09,103 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:09,103 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:09,103 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:09,103 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:09,103 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:09,103 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:09,104 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:09,104 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:09,104 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:09,104 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:09,104 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:09,104 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:09,104 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:09,104 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,104 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:09,104 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:09,105 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,105 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:09,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:09,105 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,106 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:09,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:09,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:09,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:09,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:09,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5eb76fa3[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:09,107 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:09,107 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,109 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,109 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:09,109 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:09,110 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:09,109 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,110 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:09,111 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,111 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:09,111 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:09,111 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:09,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:09,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:09,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:09,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:09,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:09,113 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:09,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:09,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:09,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:09,113 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:09,115 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,115 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:09,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,115 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:09,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:09,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:09,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,116 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:09,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:09,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,119 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,119 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,119 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:09,119 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:09,119 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:09,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:09,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:09,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:09,120 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,120 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,120 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,120 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,120 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:09,120 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:09,120 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:09,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:09,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:09,164 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:56:09,164 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:09,164 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:09,164 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:09,164 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:09,164 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:09,164 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:09,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567769179, max timestamp=1682567769192Num uniques 859; 2023-04-27 03:56:09,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:56:09,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567769179, max timestamp=1682567769192Num uniques 859; 2023-04-27 03:56:09,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:56:09,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567769179, max timestamp=1682567769192Num uniques 859; 2023-04-27 03:56:09,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:56:09,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:09,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567829207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:09,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567829208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:09,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567829209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:09,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48502 deadline: 1682567829211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,220 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:09,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:09,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:09,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:09,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:09,224 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:09,224 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:09,224 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:09,224 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:09,224 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,224 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,227 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,227 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:09,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:09,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:09,227 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,227 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:09,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:09,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:09,229 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:09,229 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:09,229 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:09,229 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:09,229 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:09,229 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:09,229 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:09,229 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:09,229 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:09,229 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:09,229 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:09,229 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:09,229 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:09,230 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,230 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:09,230 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:09,231 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:09,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:09,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,231 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:09,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:09,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:09,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:09,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:09,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@257a5949[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:09,234 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:09,234 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,235 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,235 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:09,235 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:09,235 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:09,235 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,235 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:09,241 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,241 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:09,241 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:09,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:09,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:09,241 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:09,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:09,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:09,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:09,244 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:09,244 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:09,244 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:09,244 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:09,244 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:09,246 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,246 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:09,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,246 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:09,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:09,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:09,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:09,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:09,248 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:09,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,249 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:09,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,252 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:09,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:09,252 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,252 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,252 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,252 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,252 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,252 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,252 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:09,252 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:09,252 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:09,252 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:09,253 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:09,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:09,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:09,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:09,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:09,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:09,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:09,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567829312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:09,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567829312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:09,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567829312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:09,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48502 deadline: 1682567829314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,353 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:09,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:09,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:09,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:09,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:09,356 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:09,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:09,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:09,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:09,357 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,357 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,358 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,358 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:09,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:09,358 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,358 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:09,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:09,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:09,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:09,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:09,359 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:09,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:09,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:09,360 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:09,360 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:09,360 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:09,360 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:09,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:09,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:09,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:09,360 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:09,363 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,363 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:09,363 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,363 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:09,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:09,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:09,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:09,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6125dfbe[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:09,365 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:09,365 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,366 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,366 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:09,366 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,366 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:09,366 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:09,366 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:09,368 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,368 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:09,368 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:09,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:09,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:09,368 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:09,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:09,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:09,370 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:09,370 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:09,370 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:09,370 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:09,371 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:09,372 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,372 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:09,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,372 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:09,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:09,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:09,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:09,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:09,373 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:09,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,374 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:09,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,377 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:09,377 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:09,377 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:09,377 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:09,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:09,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:09,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:09,377 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,377 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,378 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,378 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:09,378 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:09,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:09,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:09,378 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,378 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,378 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,478 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:09,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:09,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:09,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:09,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:09,481 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:09,481 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:09,481 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:09,481 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:09,482 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,482 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,483 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:09,483 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:09,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:09,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,483 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,484 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:09,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:09,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:09,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:09,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:09,484 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:09,484 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:09,485 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:09,485 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:09,485 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:09,485 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:09,485 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:09,485 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:09,485 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:09,486 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:09,486 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:09,486 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,486 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:09,486 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:09,487 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,487 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:09,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:09,487 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,487 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:09,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:09,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:09,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:09,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:09,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64953bd[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:09,489 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:09,489 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,490 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,491 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:09,491 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:09,491 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:09,491 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,491 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:09,493 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,493 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:09,493 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:09,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:09,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:09,493 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:09,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:09,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:09,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:09,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:09,495 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:09,495 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:09,495 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:09,496 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:09,496 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:09,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:09,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:09,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:09,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:09,498 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:09,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:09,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:09,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,499 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:09,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:09,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:09,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:09,502 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:09,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:09,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:09,502 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:09,502 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:09,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:09,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:09,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:09,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567829515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:09,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567829516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:09,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567829516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:09,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48502 deadline: 1682567829521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,577 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427cf41e38b1ad749bd948a892d1e38096e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427cf41e38b1ad749bd948a892d1e38096e 2023-04-27 03:56:09,581 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=210, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/8dbcd37ab3e344d690faa1de723fad83 2023-04-27 03:56:09,589 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/1b673f39b43345c0a0b9531a1af2944d 2023-04-27 03:56:09,599 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/13cc11afba3142b9ac6fe3d22b91c70d 2023-04-27 03:56:09,602 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/8dbcd37ab3e344d690faa1de723fad83 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/8dbcd37ab3e344d690faa1de723fad83 2023-04-27 03:56:09,602 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:09,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:09,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:09,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:09,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:09,606 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:09,606 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:09,606 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:09,606 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:09,606 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/8dbcd37ab3e344d690faa1de723fad83, entries=300, sequenceid=210, filesize=45.8 K 2023-04-27 03:56:09,606 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,606 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,607 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/1b673f39b43345c0a0b9531a1af2944d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/1b673f39b43345c0a0b9531a1af2944d 2023-04-27 03:56:09,608 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,608 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:09,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:09,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:09,608 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,608 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:09,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:09,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:09,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:09,609 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:09,609 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:09,609 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:09,609 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:09,609 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:09,609 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:09,610 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:09,610 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:09,610 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:09,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:09,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:09,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:09,610 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:09,611 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/1b673f39b43345c0a0b9531a1af2944d, entries=150, sequenceid=210, filesize=11.8 K 2023-04-27 03:56:09,612 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/13cc11afba3142b9ac6fe3d22b91c70d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/13cc11afba3142b9ac6fe3d22b91c70d 2023-04-27 03:56:09,613 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:09,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:09,613 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:09,613 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:09,613 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:09,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:09,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6366eea0[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:09,615 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:09,615 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,615 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/13cc11afba3142b9ac6fe3d22b91c70d, entries=150, sequenceid=210, filesize=11.8 K 2023-04-27 03:56:09,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,616 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:09,616 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,616 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:09,616 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:09,616 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:09,616 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1c25ca7c0495563171ae7ccdf0350231 in 452ms, sequenceid=210, compaction requested=true 2023-04-27 03:56:09,616 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:09,617 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:09,617 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:56:09,617 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:56:09,617 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:09,617 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:09,617 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:09,617 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 99004 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:56:09,618 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:09,618 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/A is initiating minor compaction (all files) 2023-04-27 03:56:09,618 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:09,618 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/A in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:09,618 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/9fc988a166f544ae9c2dcf6d5aa16381, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/dd922210d84c45b3a976705dc732d605, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/8dbcd37ab3e344d690faa1de723fad83] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=96.7 K 2023-04-27 03:56:09,618 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:09,618 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:56:09,618 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/B is initiating minor compaction (all files) 2023-04-27 03:56:09,618 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/B in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:09,618 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/45a54d3a10074cf9a0fe196986505ce6, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/216f8fec746d46beb3ff7e482ce64b80, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/1b673f39b43345c0a0b9531a1af2944d] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=35.6 K 2023-04-27 03:56:09,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:09,619 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 9fc988a166f544ae9c2dcf6d5aa16381, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682567765877 2023-04-27 03:56:09,619 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 45a54d3a10074cf9a0fe196986505ce6, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682567765877 2023-04-27 03:56:09,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:09,619 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 216f8fec746d46beb3ff7e482ce64b80, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1682567767013 2023-04-27 03:56:09,619 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting dd922210d84c45b3a976705dc732d605, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1682567767013 2023-04-27 03:56:09,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,620 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 1b673f39b43345c0a0b9531a1af2944d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682567769152 2023-04-27 03:56:09,620 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 8dbcd37ab3e344d690faa1de723fad83, keycount=300, bloomtype=ROW, size=45.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682567769151 2023-04-27 03:56:09,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:09,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:09,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:09,623 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:09,623 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:09,623 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:09,623 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:09,623 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:09,630 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,630 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:09,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:09,631 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#B#compaction#553 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:09,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:09,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:09,632 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:09,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:09,634 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:09,636 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#A#compaction#554 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:09,640 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,640 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,640 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,640 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,640 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,640 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:09,640 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,640 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:09,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:09,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:09,640 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:09,640 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:09,641 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:09,641 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:09,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:09,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:09,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:09,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:09,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:09,648 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/ed65f0e4f43f461e9f4d29fc38e9b636 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/ed65f0e4f43f461e9f4d29fc38e9b636 2023-04-27 03:56:09,654 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/B of 1c25ca7c0495563171ae7ccdf0350231 into ed65f0e4f43f461e9f4d29fc38e9b636(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:09,654 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:09,655 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/B, priority=13, startTime=1682567769617; duration=0sec 2023-04-27 03:56:09,655 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:09,655 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:56:09,656 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:56:09,656 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/C is initiating minor compaction (all files) 2023-04-27 03:56:09,656 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/C in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:09,656 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a566b15837784089bfd58229b241268e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a7d3b09ec7ad4e998d9939e004b7f1ef, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/13cc11afba3142b9ac6fe3d22b91c70d] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=35.6 K 2023-04-27 03:56:09,656 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting a566b15837784089bfd58229b241268e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682567765877 2023-04-27 03:56:09,657 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting a7d3b09ec7ad4e998d9939e004b7f1ef, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1682567767013 2023-04-27 03:56:09,657 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 13cc11afba3142b9ac6fe3d22b91c70d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682567769152 2023-04-27 03:56:09,661 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/08ae9c0c6a484adebb1aebd4bf38d749 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/08ae9c0c6a484adebb1aebd4bf38d749 2023-04-27 03:56:09,664 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#C#compaction#555 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:09,666 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/A of 1c25ca7c0495563171ae7ccdf0350231 into 08ae9c0c6a484adebb1aebd4bf38d749(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:09,666 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:09,666 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/A, priority=13, startTime=1682567769616; duration=0sec 2023-04-27 03:56:09,666 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:09,672 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/b2735685c24d4c9c9a43fc815dd51354 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/b2735685c24d4c9c9a43fc815dd51354 2023-04-27 03:56:09,676 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/C of 1c25ca7c0495563171ae7ccdf0350231 into b2735685c24d4c9c9a43fc815dd51354(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:09,676 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:09,676 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/C, priority=13, startTime=1682567769617; duration=0sec 2023-04-27 03:56:09,676 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:09,741 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:09,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:09,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:09,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:09,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:09,744 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:09,744 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:09,744 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:09,744 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:09,744 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,744 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,746 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:09,746 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:09,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:09,746 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,746 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:09,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:09,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:09,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:09,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:09,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:09,747 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:09,747 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:09,747 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:09,747 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:09,747 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:09,747 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:09,747 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:09,748 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:09,748 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-27 03:56:09,748 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:09,748 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:09,748 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:09,748 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:09,748 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:09,748 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:09,765 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304278dfe2c0b8d544ca7aea24a3e7931deae to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304278dfe2c0b8d544ca7aea24a3e7931deae 2023-04-27 03:56:09,770 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=249, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/0d67b9ece64e42b89ffb2432501387c1 2023-04-27 03:56:09,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:09,822 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:09,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:09,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567829828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:09,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567829829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:09,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567829830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:09,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48502 deadline: 1682567829840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:09,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567829932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:09,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567829934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:09,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:09,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567829935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:10,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567830136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:10,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567830137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:10,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567830139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,179 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/fbc923851e85401dacf54132e98fc15c 2023-04-27 03:56:10,189 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/66c5c2e0bf7947a58707ba012929c349 2023-04-27 03:56:10,193 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/0d67b9ece64e42b89ffb2432501387c1 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/0d67b9ece64e42b89ffb2432501387c1 2023-04-27 03:56:10,198 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/0d67b9ece64e42b89ffb2432501387c1, entries=150, sequenceid=249, filesize=25.3 K 2023-04-27 03:56:10,199 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/fbc923851e85401dacf54132e98fc15c as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/fbc923851e85401dacf54132e98fc15c 2023-04-27 03:56:10,202 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/fbc923851e85401dacf54132e98fc15c, entries=150, sequenceid=249, filesize=11.8 K 2023-04-27 03:56:10,202 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/66c5c2e0bf7947a58707ba012929c349 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/66c5c2e0bf7947a58707ba012929c349 2023-04-27 03:56:10,205 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/66c5c2e0bf7947a58707ba012929c349, entries=150, sequenceid=249, filesize=11.8 K 2023-04-27 03:56:10,206 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 1c25ca7c0495563171ae7ccdf0350231 in 458ms, sequenceid=249, compaction requested=false 2023-04-27 03:56:10,206 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:10,206 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:10,206 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:10,206 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:10,206 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,206 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:10,206 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:10,208 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,208 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:10,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:10,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:10,209 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:10,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:10,209 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:10,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:10,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:10,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:10,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14215394[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:10,210 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:10,210 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:10,213 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:10,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:10,213 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:10,213 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:10,213 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:10,213 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,213 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:10,214 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:10,214 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:10,214 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,214 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:10,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:10,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:10,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:10,217 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:10,217 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:10,217 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:10,217 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:10,217 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:10,219 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:10,219 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:10,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:10,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:10,219 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:10,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:10,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:10,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:10,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:10,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:10,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:10,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:10,219 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:10,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:10,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:10,221 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:10,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:10,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,223 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,223 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:10,223 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:10,223 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:10,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:10,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:10,223 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:10,223 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,224 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:10,224 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:10,224 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:10,223 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:10,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:10,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:10,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:10,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:10,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:10,224 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:10,224 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:10,324 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:10,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:10,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:10,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:10,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:10,328 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:10,328 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:10,328 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:10,328 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:10,328 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:10,328 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:10,331 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:10,331 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:10,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:10,331 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,331 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:10,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:10,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:10,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:10,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:10,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:10,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:10,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:10,332 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:10,332 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:10,332 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:10,332 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:10,332 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:10,332 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:10,332 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:10,332 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:56:10,333 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:10,333 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:10,333 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:10,333 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:10,333 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:10,333 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:10,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:10,349 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:10,352 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427e69f72756a8147e99072dba6f8dc61ff to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427e69f72756a8147e99072dba6f8dc61ff 2023-04-27 03:56:10,357 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=259, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/06bf6d3c060a406cbb8d9ab36841ed9c 2023-04-27 03:56:10,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567770388, max timestamp=1682567770398Num uniques 859; 2023-04-27 03:56:10,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:56:10,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567770388, max timestamp=1682567770398Num uniques 859; 2023-04-27 03:56:10,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:56:10,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567770388, max timestamp=1682567770398Num uniques 859; 2023-04-27 03:56:10,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:56:10,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:10,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567830441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:10,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567830444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:10,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567830445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:10,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567830445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:10,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48474 deadline: 1682567830519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,523 DEBUG [Thread-4053] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8173 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., hostname=jenkins-hbase4.apache.org,40885,1682567618305, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 03:56:10,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:10,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567830549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:10,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567830550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:10,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567830550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:10,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567830753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:10,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567830753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:10,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567830754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:10,766 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/ea8543f0e7064da48b05e4ad2838c81d 2023-04-27 03:56:10,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:10,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567830946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:11,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567831057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:11,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567831057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:11,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567831059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,177 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/04b9cb9d75a34d1c933778ed0cf1585f 2023-04-27 03:56:11,183 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/06bf6d3c060a406cbb8d9ab36841ed9c as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/06bf6d3c060a406cbb8d9ab36841ed9c 2023-04-27 03:56:11,186 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/06bf6d3c060a406cbb8d9ab36841ed9c, entries=150, sequenceid=259, filesize=25.4 K 2023-04-27 03:56:11,187 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/ea8543f0e7064da48b05e4ad2838c81d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/ea8543f0e7064da48b05e4ad2838c81d 2023-04-27 03:56:11,191 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/ea8543f0e7064da48b05e4ad2838c81d, entries=150, sequenceid=259, filesize=11.9 K 2023-04-27 03:56:11,191 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/04b9cb9d75a34d1c933778ed0cf1585f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/04b9cb9d75a34d1c933778ed0cf1585f 2023-04-27 03:56:11,194 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/04b9cb9d75a34d1c933778ed0cf1585f, entries=150, sequenceid=259, filesize=11.9 K 2023-04-27 03:56:11,195 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1c25ca7c0495563171ae7ccdf0350231 in 863ms, sequenceid=259, compaction requested=true 2023-04-27 03:56:11,195 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:11,195 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:11,195 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:11,195 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:11,195 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,195 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:11,195 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:11,197 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:11,197 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:11,197 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,197 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:11,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:11,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b36a681[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:11,198 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:11,198 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,199 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,200 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:11,200 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:11,200 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,200 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:11,200 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:11,201 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,201 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:11,201 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:11,201 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:11,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:11,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:11,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:11,203 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:11,203 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:11,203 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:11,203 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:11,203 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:11,206 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,206 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:11,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,207 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:11,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:11,207 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,208 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,211 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,211 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:11,211 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:11,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:11,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:11,211 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:11,211 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:11,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:11,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:11,211 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:11,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:11,211 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:11,211 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:11,211 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,212 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,212 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,212 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,312 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:11,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:11,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:11,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:11,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:11,314 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:11,315 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:11,315 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:11,315 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:11,315 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,315 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,316 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,316 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:11,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:11,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:11,316 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,317 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:11,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:11,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:11,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:11,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:11,318 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:11,318 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:11,318 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:11,318 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:11,318 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:11,318 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:11,318 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:11,318 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-27 03:56:11,318 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:11,318 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:11,318 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:11,318 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:11,318 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:11,318 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:11,334 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427da0893d87860438ebaa4c717a522616c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427da0893d87860438ebaa4c717a522616c 2023-04-27 03:56:11,349 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=296, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/be9ea24a70f24feba5e4af74bae66136 2023-04-27 03:56:11,359 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/b86fb1287bb843f7ad6b6af11e275270 2023-04-27 03:56:11,381 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/57e0aab6b32546f4ad139a085c0f5e4e 2023-04-27 03:56:11,385 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/be9ea24a70f24feba5e4af74bae66136 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/be9ea24a70f24feba5e4af74bae66136 2023-04-27 03:56:11,388 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/be9ea24a70f24feba5e4af74bae66136, entries=150, sequenceid=296, filesize=25.4 K 2023-04-27 03:56:11,389 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/b86fb1287bb843f7ad6b6af11e275270 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/b86fb1287bb843f7ad6b6af11e275270 2023-04-27 03:56:11,393 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/b86fb1287bb843f7ad6b6af11e275270, entries=150, sequenceid=296, filesize=11.9 K 2023-04-27 03:56:11,393 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/57e0aab6b32546f4ad139a085c0f5e4e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/57e0aab6b32546f4ad139a085c0f5e4e 2023-04-27 03:56:11,397 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/57e0aab6b32546f4ad139a085c0f5e4e, entries=150, sequenceid=296, filesize=11.9 K 2023-04-27 03:56:11,397 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 1c25ca7c0495563171ae7ccdf0350231 in 79ms, sequenceid=296, compaction requested=true 2023-04-27 03:56:11,397 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:11,397 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:11,398 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:11,398 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:11,398 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,398 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:11,398 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:11,399 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,399 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:11,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:11,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:11,400 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,400 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:11,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:11,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:11,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:11,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17aed2e1[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:11,401 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:11,401 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,403 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,403 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,403 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:11,403 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:11,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:11,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:11,405 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,405 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:11,405 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:11,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:11,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:11,405 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:11,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:11,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:11,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:11,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:11,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:11,408 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:11,408 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:11,408 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:11,408 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:11,408 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:11,410 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,410 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:11,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:11,410 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:11,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:11,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:11,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:11,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,411 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:11,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,412 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:11,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,415 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:11,415 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:11,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:11,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:11,415 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:11,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:11,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:11,415 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:11,415 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,415 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:11,415 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,415 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:11,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:11,415 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:11,416 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,416 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,416 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,516 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:11,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:11,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:11,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:11,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:11,519 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:11,519 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:11,519 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:11,519 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:11,519 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,520 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,521 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,521 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:11,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:11,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:11,521 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,521 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:11,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:11,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:11,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:11,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:11,522 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:11,522 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:11,522 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:11,522 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:11,522 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:11,522 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:11,523 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:11,523 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:11,523 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:11,523 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:11,523 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:11,523 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,523 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:11,523 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:11,524 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,524 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:11,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:11,525 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:11,525 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:11,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:11,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:11,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:11,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43693d5a[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:11,526 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:11,526 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,527 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,528 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:11,528 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:11,528 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,528 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:11,528 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:11,530 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:11,530 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:11,530 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,530 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:11,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:11,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:11,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:11,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:11,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:11,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:11,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:11,533 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:11,533 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:11,533 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:11,533 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:11,533 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:11,534 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,534 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:11,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:11,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,534 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:11,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:11,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:11,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:11,535 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:11,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,536 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:11,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,539 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,539 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,539 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,539 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:11,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:11,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:11,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:11,539 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:11,539 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:11,539 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:11,539 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:11,539 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:11,540 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,540 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:11,540 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:11,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:11,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:11,589 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:56:11,590 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:11,590 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:11,590 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:11,590 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:11,590 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:11,590 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:11,602 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042726c9cb05cb29402ebb8c344508b00669 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042726c9cb05cb29402ebb8c344508b00669 2023-04-27 03:56:11,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567771612, max timestamp=1682567771616Num uniques 859; 2023-04-27 03:56:11,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:11,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:56:11,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:11,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567771612, max timestamp=1682567771616Num uniques 859; 2023-04-27 03:56:11,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567831614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567771612, max timestamp=1682567771616Num uniques 859; 2023-04-27 03:56:11,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48458 deadline: 1682567831614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:56:11,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:56:11,640 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:11,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:11,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:11,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:11,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:11,643 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:11,643 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:11,643 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:11,643 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:11,644 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,644 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,645 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:11,645 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:11,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:11,645 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,645 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:11,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:11,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:11,646 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:11,646 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:11,646 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:11,646 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:11,646 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:11,646 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:11,646 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:11,647 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:11,647 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:11,647 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:11,647 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:11,647 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:11,647 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:11,647 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,647 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:11,647 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:11,650 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,650 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:11,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:11,650 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,650 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:11,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:11,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:11,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:11,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:11,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dca7b50[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:11,651 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:11,651 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,652 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,652 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:11,652 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:11,652 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:11,653 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,653 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:11,654 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:11,654 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,654 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:11,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:11,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:11,654 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:11,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:11,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:11,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:11,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:11,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:11,656 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:11,656 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:11,656 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:11,656 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:11,656 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:11,658 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,658 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,658 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:11,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:11,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:11,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:11,658 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:11,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:11,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:11,659 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:11,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,663 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:11,663 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:11,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:11,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:11,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:11,664 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,664 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:11,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:11,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:11,664 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:11,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:11,664 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,664 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:11,664 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,664 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:11,664 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,664 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,664 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,685 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:11,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567831682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:11,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567831719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:11,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567831719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,764 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:11,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:11,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:11,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:11,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:11,767 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:11,767 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:11,768 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:11,768 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:11,768 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,768 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,769 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:11,769 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:11,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:11,769 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,769 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:11,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:11,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:11,770 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:11,770 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:11,770 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:11,770 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:11,770 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:11,771 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:11,771 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:11,771 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:11,771 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:11,771 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:11,771 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:11,771 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:11,771 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:11,771 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,771 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:11,771 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:11,773 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:11,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:11,773 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:11,774 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,774 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:11,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:11,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:11,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:11,775 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:11,775 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33a6dcc6[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:11,776 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,776 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:11,776 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:11,776 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:11,776 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,776 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:11,778 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,778 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:11,778 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:11,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:11,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:11,778 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:11,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:11,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:11,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:11,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:11,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:11,780 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:11,780 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:11,780 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:11,780 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:11,780 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:11,782 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,782 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,782 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:11,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:11,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:11,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:11,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:11,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:11,782 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:11,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,783 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:11,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:11,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567831786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,788 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,788 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,788 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,788 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:11,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:11,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:11,788 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:11,788 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:11,788 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:11,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:11,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:11,788 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,788 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:11,788 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,788 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:11,788 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:11,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:11,888 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:11,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:11,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:11,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:11,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:11,892 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:11,892 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:11,892 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:11,892 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:11,892 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,892 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,893 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,893 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:11,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:11,894 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,894 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:11,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:11,895 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:11,895 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:11,895 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:11,895 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:11,895 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:11,895 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:11,895 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:11,895 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:11,895 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:11,896 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:11,896 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:11,896 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:11,896 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,896 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:11,896 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:11,897 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,897 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:11,898 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,898 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:11,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:11,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@527e3989[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:11,899 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:11,899 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,901 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,901 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:11,901 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:11,901 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,901 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:11,901 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:11,902 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:11,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:11,903 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:11,903 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:11,903 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:11,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:11,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:11,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:11,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:11,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:11,905 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:11,905 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:11,905 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:11,905 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:11,905 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:11,906 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,906 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,906 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:11,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:11,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:11,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:11,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:11,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:11,907 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:11,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,909 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:11,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:11,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:11,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:11,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:11,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:11,911 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:11,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:11,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:11,911 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:11,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:11,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:11,911 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:11,912 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:11,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:11,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:11,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:11,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567831922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:11,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567831922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,954 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:11,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567831953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:11,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:11,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567831990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,009 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=308, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/fb2aaba0a5f34d469df22d4ff9ceca87 2023-04-27 03:56:12,012 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:12,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:12,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:12,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:12,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:12,015 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:12,015 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:12,015 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:12,015 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:12,016 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,016 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,023 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,023 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:12,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:12,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:12,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,023 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,024 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:12,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:12,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:12,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:12,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:12,024 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:12,024 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:12,024 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:12,025 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:12,025 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:12,025 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:12,025 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:12,025 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:12,025 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:12,025 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:12,025 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:12,025 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,025 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:12,025 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:12,027 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,027 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:12,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:12,027 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,027 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:12,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:12,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:12,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:12,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:12,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58f74d71[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:12,029 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:12,029 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,030 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,030 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:12,030 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:12,030 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:12,031 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,031 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:12,032 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,032 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:12,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:12,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:12,032 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:12,032 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:12,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:12,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:12,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:12,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:12,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:12,034 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:12,034 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:12,034 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:12,034 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:12,034 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:12,037 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,037 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:12,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:12,037 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:12,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:12,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:12,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:12,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,038 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:12,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:12,040 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,042 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,042 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,042 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:12,042 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:12,042 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:12,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:12,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:12,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:12,042 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:12,042 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:12,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:12,043 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,043 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,043 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,043 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:12,043 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:12,143 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:12,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:12,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:12,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:12,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:12,147 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:12,147 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:12,147 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:12,147 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:12,147 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,147 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,148 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,148 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:12,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:12,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:12,149 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,149 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:12,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:12,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:12,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:12,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:12,150 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:12,150 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:12,150 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:12,150 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:12,150 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:12,150 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:12,150 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:12,150 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:12,150 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:12,150 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:12,150 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:12,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:12,151 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:12,152 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,152 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:12,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:12,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:12,153 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,153 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:12,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:12,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:12,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:12,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56e9bdc4[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:12,154 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:12,155 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,157 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,157 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:12,157 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:12,157 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,157 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:12,157 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:12,159 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,159 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:12,159 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:12,159 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:12,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:12,161 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:12,161 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:12,161 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:12,161 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:12,161 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:12,163 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,163 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:12,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,163 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:12,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:12,164 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:12,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,165 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:12,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,168 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,168 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,168 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,168 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,168 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,168 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:12,168 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:12,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:12,168 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:12,168 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,168 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:12,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:12,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:12,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:12,182 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:12,182 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:12,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:12,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:12,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:12,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:12,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48534 deadline: 1682567832226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567832227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,282 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:12,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:12,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:12,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:12,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:12,285 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:12,285 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:12,285 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:12,285 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:12,286 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,286 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,288 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:12,288 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:12,289 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,289 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:12,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,290 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:12,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:12,290 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:12,290 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:12,290 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:12,290 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:12,290 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:12,290 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:12,290 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:12,291 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:12,291 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:12,291 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:12,291 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:12,291 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,291 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:12,291 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:12,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:12,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567832293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,295 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,295 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:12,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:12,295 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,295 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:12,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:12,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:12,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:12,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:12,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53ac0b33[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:12,298 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:12,298 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,299 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,299 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:12,299 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:12,299 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:12,299 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,299 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:12,301 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:12,301 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:12,301 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,301 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:12,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:12,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:12,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:12,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:12,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:12,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:12,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:12,303 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:12,303 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:12,303 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:12,303 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:12,303 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:12,305 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,306 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:12,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:12,306 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:12,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:12,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:12,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:12,307 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:12,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,308 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:12,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,310 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:12,310 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:12,310 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,310 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,310 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,310 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,310 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,310 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,310 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:12,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:12,310 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:12,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:12,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:12,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:12,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:12,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:12,311 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:12,311 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:12,411 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:12,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:12,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:12,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:12,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:12,414 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:12,414 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:12,414 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:12,414 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:12,415 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,415 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,416 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:12,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:12,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:12,416 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,416 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,416 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:12,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:12,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:12,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:12,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:12,417 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:12,417 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:12,418 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:12,418 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:12,418 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:12,418 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:12,418 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:12,418 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:12,418 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:12,418 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:12,418 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:12,418 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,418 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:12,418 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:12,419 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:12,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:12,420 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=308 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/31bd92da3cb5413ea1ddd1d4d36f9122 2023-04-27 03:56:12,420 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:12,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:12,421 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,421 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:12,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:12,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:12,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@703dd2be[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:12,422 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:12,422 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,424 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,424 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:12,424 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:12,424 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:12,424 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,424 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:12,426 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:12,426 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,426 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:12,426 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:12,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:12,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:12,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:12,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:12,429 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:12,429 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:12,429 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:12,429 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:12,429 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:12,429 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=308 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/7745c74e0a96457d85ac6386ac2d370d 2023-04-27 03:56:12,430 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,430 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:12,430 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:12,431 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:12,433 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/fb2aaba0a5f34d469df22d4ff9ceca87 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/fb2aaba0a5f34d469df22d4ff9ceca87 2023-04-27 03:56:12,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,433 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:12,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,436 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:12,436 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:12,437 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:12,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:12,437 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,437 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,437 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,437 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,437 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,437 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:12,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:12,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:12,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:12,437 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/fb2aaba0a5f34d469df22d4ff9ceca87, entries=400, sequenceid=308, filesize=59.8 K 2023-04-27 03:56:12,437 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:12,438 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:12,438 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:12,438 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/31bd92da3cb5413ea1ddd1d4d36f9122 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/31bd92da3cb5413ea1ddd1d4d36f9122 2023-04-27 03:56:12,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:12,441 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/31bd92da3cb5413ea1ddd1d4d36f9122, entries=150, sequenceid=308, filesize=11.9 K 2023-04-27 03:56:12,442 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/7745c74e0a96457d85ac6386ac2d370d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/7745c74e0a96457d85ac6386ac2d370d 2023-04-27 03:56:12,445 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/7745c74e0a96457d85ac6386ac2d370d, entries=150, sequenceid=308, filesize=11.9 K 2023-04-27 03:56:12,446 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 1c25ca7c0495563171ae7ccdf0350231 in 857ms, sequenceid=308, compaction requested=true 2023-04-27 03:56:12,446 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:12,446 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:12,446 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:12,446 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 03:56:12,446 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 03:56:12,446 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:12,447 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61024 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 03:56:12,447 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/B is initiating minor compaction (all files) 2023-04-27 03:56:12,447 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/B in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:12,447 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/ed65f0e4f43f461e9f4d29fc38e9b636, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/fbc923851e85401dacf54132e98fc15c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/ea8543f0e7064da48b05e4ad2838c81d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/b86fb1287bb843f7ad6b6af11e275270, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/31bd92da3cb5413ea1ddd1d4d36f9122] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=59.6 K 2023-04-27 03:56:12,448 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 165532 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 03:56:12,448 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/A is initiating minor compaction (all files) 2023-04-27 03:56:12,448 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/A in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:12,448 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/08ae9c0c6a484adebb1aebd4bf38d749, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/0d67b9ece64e42b89ffb2432501387c1, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/06bf6d3c060a406cbb8d9ab36841ed9c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/be9ea24a70f24feba5e4af74bae66136, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/fb2aaba0a5f34d469df22d4ff9ceca87] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=161.7 K 2023-04-27 03:56:12,448 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting ed65f0e4f43f461e9f4d29fc38e9b636, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682567769152 2023-04-27 03:56:12,448 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 08ae9c0c6a484adebb1aebd4bf38d749, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682567769152 2023-04-27 03:56:12,448 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting fbc923851e85401dacf54132e98fc15c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1682567769207 2023-04-27 03:56:12,448 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 0d67b9ece64e42b89ffb2432501387c1, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1682567769207 2023-04-27 03:56:12,449 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting ea8543f0e7064da48b05e4ad2838c81d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1682567769826 2023-04-27 03:56:12,450 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 06bf6d3c060a406cbb8d9ab36841ed9c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1682567769826 2023-04-27 03:56:12,450 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting b86fb1287bb843f7ad6b6af11e275270, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1682567770426 2023-04-27 03:56:12,450 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting be9ea24a70f24feba5e4af74bae66136, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1682567770426 2023-04-27 03:56:12,450 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 31bd92da3cb5413ea1ddd1d4d36f9122, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1682567771575 2023-04-27 03:56:12,450 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting fb2aaba0a5f34d469df22d4ff9ceca87, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1682567771575 2023-04-27 03:56:12,457 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#B#compaction#568 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:12,468 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#A#compaction#569 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:12,469 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/998e4efe5a1742a9998a6135dbaf037d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/998e4efe5a1742a9998a6135dbaf037d 2023-04-27 03:56:12,473 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/B of 1c25ca7c0495563171ae7ccdf0350231 into 998e4efe5a1742a9998a6135dbaf037d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:12,473 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:12,473 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/B, priority=11, startTime=1682567772446; duration=0sec 2023-04-27 03:56:12,473 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:12,473 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-27 03:56:12,474 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60990 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-27 03:56:12,474 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/C is initiating minor compaction (all files) 2023-04-27 03:56:12,474 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/C in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:12,474 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/b2735685c24d4c9c9a43fc815dd51354, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/66c5c2e0bf7947a58707ba012929c349, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/04b9cb9d75a34d1c933778ed0cf1585f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/57e0aab6b32546f4ad139a085c0f5e4e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/7745c74e0a96457d85ac6386ac2d370d] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=59.6 K 2023-04-27 03:56:12,475 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting b2735685c24d4c9c9a43fc815dd51354, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682567769152 2023-04-27 03:56:12,475 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 66c5c2e0bf7947a58707ba012929c349, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1682567769207 2023-04-27 03:56:12,475 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 04b9cb9d75a34d1c933778ed0cf1585f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1682567769826 2023-04-27 03:56:12,475 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 57e0aab6b32546f4ad139a085c0f5e4e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1682567770426 2023-04-27 03:56:12,476 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 7745c74e0a96457d85ac6386ac2d370d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1682567771575 2023-04-27 03:56:12,485 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#C#compaction#570 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:12,538 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:12,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:12,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:12,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:12,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:12,541 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:12,541 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:12,541 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:12,541 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:12,541 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,541 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,543 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,543 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:12,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:12,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:12,543 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,543 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,544 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:12,544 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:12,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:12,544 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:12,544 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:12,544 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:12,545 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:12,545 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:12,545 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:12,545 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:12,545 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-27 03:56:12,545 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:12,545 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:12,545 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:12,545 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:12,545 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:12,545 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:12,566 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427b645897874e64074a37bc94f7170ffa2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427b645897874e64074a37bc94f7170ffa2 2023-04-27 03:56:12,571 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=345, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/29dca4d280b04757be7e16e7f1cedab9 2023-04-27 03:56:12,580 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/b07ea9d691bc45bb91a97c337c415045 2023-04-27 03:56:12,588 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/9306df8ba8af40139873762d875c48d0 2023-04-27 03:56:12,591 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/29dca4d280b04757be7e16e7f1cedab9 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/29dca4d280b04757be7e16e7f1cedab9 2023-04-27 03:56:12,594 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/29dca4d280b04757be7e16e7f1cedab9, entries=150, sequenceid=345, filesize=25.4 K 2023-04-27 03:56:12,595 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/b07ea9d691bc45bb91a97c337c415045 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/b07ea9d691bc45bb91a97c337c415045 2023-04-27 03:56:12,602 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/b07ea9d691bc45bb91a97c337c415045, entries=150, sequenceid=345, filesize=11.9 K 2023-04-27 03:56:12,602 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/9306df8ba8af40139873762d875c48d0 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/9306df8ba8af40139873762d875c48d0 2023-04-27 03:56:12,613 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/9306df8ba8af40139873762d875c48d0, entries=150, sequenceid=345, filesize=11.9 K 2023-04-27 03:56:12,614 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for 1c25ca7c0495563171ae7ccdf0350231 in 69ms, sequenceid=345, compaction requested=false 2023-04-27 03:56:12,614 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:12,614 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:12,614 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:12,615 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:12,615 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,615 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:12,615 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:12,616 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:12,616 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:12,617 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,617 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:12,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:12,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:12,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:12,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12c3950d[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:12,618 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:12,618 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,619 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,619 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:12,619 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:12,619 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:12,619 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,620 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:12,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:12,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:12,621 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:12,620 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:12,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:12,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:12,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:12,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:12,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:12,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:12,623 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:12,623 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:12,623 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:12,623 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:12,623 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:12,626 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,626 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,626 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:12,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:12,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:12,627 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:12,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,628 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:12,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,630 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:12,630 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:12,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:12,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:12,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:12,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:12,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:12,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:12,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,631 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:12,631 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:12,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:12,631 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:12,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,731 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:12,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:12,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:12,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:12,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:12,735 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:12,735 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:12,735 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:12,735 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:12,735 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,735 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,736 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:12,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:12,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,737 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:12,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:12,738 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:12,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:12,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:12,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:12,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:12,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:12,738 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:12,738 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:12,738 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:12,739 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:12,739 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:12,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:12,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:12,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:12,739 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:12,740 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,740 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:12,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:12,740 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,740 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:12,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:12,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:12,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:12,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:12,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@95f1d0c[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:12,742 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:12,742 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,750 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:12,750 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:12,750 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:12,750 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,750 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:12,751 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:12,751 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:12,751 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:12,751 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:12,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:12,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:12,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:12,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:12,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:12,753 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:12,753 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:12,753 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:12,753 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:12,754 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:12,755 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,755 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:12,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:12,755 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:12,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,756 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,757 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,762 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:12,762 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:12,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:12,762 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,762 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,762 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,762 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,762 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,762 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,763 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:12,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:12,763 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:12,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:12,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:12,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:12,763 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:12,763 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:12,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:12,772 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:56:12,772 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:12,772 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:12,772 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:12,772 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:12,772 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:12,772 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:12,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:12,808 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682567772797, max timestamp=1682567772806Num uniques 859; 2023-04-27 03:56:12,808 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682567772797, max timestamp=1682567772806Num uniques 859; 2023-04-27 03:56:12,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567832805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,808 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682567772797, max timestamp=1682567772806Num uniques 859; 2023-04-27 03:56:12,808 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-27 03:56:12,808 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-27 03:56:12,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-27 03:56:12,841 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:12,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567832837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:12,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567832841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,863 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:12,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:12,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:12,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:12,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:12,866 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:12,866 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:12,866 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:12,867 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:12,867 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,867 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,868 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:12,868 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:12,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:12,868 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,869 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:12,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,869 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:12,869 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:12,869 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:12,869 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:12,869 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:12,870 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:12,870 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:12,870 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:12,870 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:12,870 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:12,870 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:12,870 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:12,870 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:12,870 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:12,870 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,870 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:12,870 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:12,871 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,872 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:12,872 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,872 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:12,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:12,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a197a87[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:12,873 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:12,873 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,874 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,874 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:12,874 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:12,875 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:12,875 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,875 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:12,877 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,877 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:12,877 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:12,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:12,877 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:12,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:12,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:12,880 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:12,880 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:12,880 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:12,880 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:12,880 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:12,881 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,881 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:12,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:12,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:12,882 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:12,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:12,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:12,883 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,884 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,886 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,886 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:12,886 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,886 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,886 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:12,886 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:12,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:12,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:12,886 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:12,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:12,886 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,886 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:12,886 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:12,886 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,886 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:12,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:12,902 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/46bb7ae5d5e848c6bd202d79246b1575 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/46bb7ae5d5e848c6bd202d79246b1575 2023-04-27 03:56:12,902 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/a34a4d1591b5423d99d50884dd85855e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a34a4d1591b5423d99d50884dd85855e 2023-04-27 03:56:12,906 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/C of 1c25ca7c0495563171ae7ccdf0350231 into a34a4d1591b5423d99d50884dd85855e(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:12,906 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/A of 1c25ca7c0495563171ae7ccdf0350231 into 46bb7ae5d5e848c6bd202d79246b1575(size=26.0 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:12,906 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:12,906 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:12,906 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/C, priority=11, startTime=1682567772446; duration=0sec 2023-04-27 03:56:12,906 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/A, priority=11, startTime=1682567772446; duration=0sec 2023-04-27 03:56:12,906 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:12,906 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:12,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:12,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567832909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,945 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:12,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567832942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:12,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567832945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,986 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:12,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:12,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:12,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:12,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:12,990 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:12,990 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:12,990 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:12,990 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:12,990 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,990 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,991 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,992 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:12,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:12,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:12,992 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,992 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:12,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:12,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:12,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:12,993 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:12,993 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:12,993 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:12,993 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:12,993 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:12,993 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:12,993 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:12,993 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:12,994 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:12,994 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:12,994 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:12,994 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:12,994 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:12,994 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:12,994 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,994 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:12,994 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:12,997 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,997 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:12,997 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:12,997 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:12,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:12,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:12,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:12,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24afe758[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:12,999 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:13,000 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,001 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,001 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:13,001 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:13,001 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:13,001 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,001 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:13,002 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:13,002 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:13,002 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,003 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:13,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:13,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:13,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:13,005 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:13,005 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:13,005 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:13,005 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:13,005 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:13,007 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,007 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:13,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:13,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,007 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:13,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:13,008 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:13,009 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:13,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,012 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:13,012 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,013 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,013 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:13,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:13,012 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:13,013 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,013 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,013 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,013 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,013 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:13,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:13,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:13,013 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:13,013 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:13,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:13,113 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:13,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:13,114 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:13,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567833113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:13,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:13,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:13,116 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:13,116 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:13,116 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:13,116 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:13,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,117 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,118 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:13,118 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:13,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:13,118 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,118 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:13,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:13,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:13,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:13,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:13,119 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:13,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:13,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:13,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:13,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:13,120 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:13,120 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:13,120 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:13,120 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:13,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:13,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:13,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:13,120 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:13,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,121 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:13,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:13,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:13,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:13,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:13,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:13,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:13,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a4a653d[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:13,123 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:13,123 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,124 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,124 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:13,124 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:13,124 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:13,124 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,124 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:13,126 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,126 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:13,126 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:13,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:13,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:13,126 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:13,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:13,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:13,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:13,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:13,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:13,128 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:13,128 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:13,128 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:13,128 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:13,128 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:13,132 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,132 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:13,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:13,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,132 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:13,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:13,133 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:13,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:13,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:13,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,134 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:13,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,136 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,136 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,136 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:13,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:13,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:13,136 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:13,136 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:13,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:13,136 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:13,136 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:13,136 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:13,136 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,136 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,136 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,136 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:13,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:13,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:13,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567833147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:13,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567833149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,185 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427a1333c6a3247463b830ad044b0447cfd to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427a1333c6a3247463b830ad044b0447cfd 2023-04-27 03:56:13,236 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:13,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:13,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:13,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:13,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:13,239 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:13,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:13,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:13,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:13,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,241 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:13,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:13,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:13,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,241 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:13,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:13,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,242 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:13,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:13,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:13,242 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:13,242 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:13,242 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:13,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:13,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:13,243 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:13,243 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:13,243 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:13,243 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:13,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:13,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:13,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:13,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:13,245 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,245 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:13,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:13,245 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,245 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:13,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:13,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:13,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:13,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:13,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b0caa05[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:13,247 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:13,247 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,248 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,248 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:13,248 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:13,248 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:13,248 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,248 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:13,249 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,249 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:13,249 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:13,249 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:13,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:13,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:13,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:13,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:13,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:13,252 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:13,252 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:13,252 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:13,252 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:13,252 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:13,254 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,254 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,254 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:13,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:13,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:13,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,254 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:13,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:13,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:13,255 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,258 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,258 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,258 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,258 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:13,258 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:13,258 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:13,259 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:13,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:13,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:13,259 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:13,259 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,259 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:13,259 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:13,259 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:13,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:13,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:13,359 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:13,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:13,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:13,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:13,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:13,363 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:13,363 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:13,363 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:13,363 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:13,363 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,363 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,364 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:13,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:13,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:13,364 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,365 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,365 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:13,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:13,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:13,366 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:13,366 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:13,366 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:13,366 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:13,366 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:13,366 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:13,366 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:13,366 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:13,366 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:13,366 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:13,366 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:13,366 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:13,366 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:13,366 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,366 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:13,366 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:13,368 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,368 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:13,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:13,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:13,368 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,368 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:13,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:13,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ab03ea1[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:13,370 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:13,370 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,371 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,371 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:13,371 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:13,371 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:13,371 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,371 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:13,373 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:13,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:13,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:13,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:13,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:13,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:13,375 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:13,375 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:13,375 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:13,375 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:13,375 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:13,377 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:13,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:13,377 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,377 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:13,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:13,377 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:13,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:13,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:13,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,378 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:13,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,380 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,381 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,381 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,381 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:13,381 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,381 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,381 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:13,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:13,381 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:13,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:13,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:13,381 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:13,381 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:13,381 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:13,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:13,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:13,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:13,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:13,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567833416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:13,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567833451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:13,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567833453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,481 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:13,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:13,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:13,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:13,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:13,485 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:13,485 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:13,485 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:13,485 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:13,485 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,485 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,486 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,486 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:13,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:13,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:13,487 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,487 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:13,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:13,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:13,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:13,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:13,488 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:13,488 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:13,488 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:13,488 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:13,488 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:13,488 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:13,488 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:13,488 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:13,488 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:13,488 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:13,488 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:13,489 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,489 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:13,489 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:13,491 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,491 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:13,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:13,492 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,492 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:13,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:13,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:13,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:13,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:13,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71fe9988[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:13,493 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:13,493 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,494 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,494 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:13,494 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:13,494 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:13,494 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,494 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:13,495 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:13,495 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:13,495 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,496 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:13,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:13,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:13,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:13,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:13,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:13,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:13,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:13,499 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:13,499 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:13,499 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:13,499 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:13,499 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:13,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:13,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,501 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:13,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:13,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:13,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:13,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,503 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:13,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,504 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:13,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,509 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,509 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,509 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:13,509 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:13,509 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:13,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:13,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:13,509 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,509 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:13,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:13,509 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,510 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,510 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:13,510 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:13,510 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:13,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:13,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:13,596 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=357, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/23fc139379e54279a94ce7664e467a1e 2023-04-27 03:56:13,605 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/7dd616c47e23491597947e082e84eb6e 2023-04-27 03:56:13,610 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:13,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:13,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:13,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:13,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:13,616 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:13,616 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:13,616 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:13,616 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:13,617 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,617 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,618 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/ad73db38881f4483aec0437c9cca4d6a 2023-04-27 03:56:13,618 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:13,618 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:13,619 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,619 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:13,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,620 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:13,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:13,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:13,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:13,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:13,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:13,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:13,620 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:13,620 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:13,620 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:13,620 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:13,620 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:13,621 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:13,621 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:13,621 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,621 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:13,621 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:13,622 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,622 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:13,623 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,623 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:13,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:13,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:13,623 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/23fc139379e54279a94ce7664e467a1e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/23fc139379e54279a94ce7664e467a1e 2023-04-27 03:56:13,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:13,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:13,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8e47907[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:13,624 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:13,625 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,626 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:13,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:13,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:13,628 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/23fc139379e54279a94ce7664e467a1e, entries=200, sequenceid=357, filesize=32.3 K 2023-04-27 03:56:13,628 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,628 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:13,628 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:13,628 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:13,629 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:13,629 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:13,629 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:13,629 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:13,629 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:13,629 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:13,629 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:13,629 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/7dd616c47e23491597947e082e84eb6e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/7dd616c47e23491597947e082e84eb6e 2023-04-27 03:56:13,630 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:13,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:13,630 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:13,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:13,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:13,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:13,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,631 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:13,632 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/7dd616c47e23491597947e082e84eb6e, entries=150, sequenceid=357, filesize=11.9 K 2023-04-27 03:56:13,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,633 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/ad73db38881f4483aec0437c9cca4d6a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/ad73db38881f4483aec0437c9cca4d6a 2023-04-27 03:56:13,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:13,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:13,634 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,636 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,636 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:13,636 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:13,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:13,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:13,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:13,636 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,636 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,636 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,636 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:13,636 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,636 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:13,636 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:13,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:13,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:13,636 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:13,636 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:13,636 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/ad73db38881f4483aec0437c9cca4d6a, entries=150, sequenceid=357, filesize=11.9 K 2023-04-27 03:56:13,637 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 1c25ca7c0495563171ae7ccdf0350231 in 865ms, sequenceid=357, compaction requested=true 2023-04-27 03:56:13,637 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:13,637 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:13,637 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:13,637 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:13,637 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:56:13,637 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:56:13,638 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:56:13,638 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85792 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:56:13,638 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/B is initiating minor compaction (all files) 2023-04-27 03:56:13,638 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/A is initiating minor compaction (all files) 2023-04-27 03:56:13,638 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/B in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:13,638 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/A in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:13,638 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/998e4efe5a1742a9998a6135dbaf037d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/b07ea9d691bc45bb91a97c337c415045, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/7dd616c47e23491597947e082e84eb6e] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=36.3 K 2023-04-27 03:56:13,638 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/46bb7ae5d5e848c6bd202d79246b1575, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/29dca4d280b04757be7e16e7f1cedab9, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/23fc139379e54279a94ce7664e467a1e] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=83.8 K 2023-04-27 03:56:13,639 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 46bb7ae5d5e848c6bd202d79246b1575, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1682567771575 2023-04-27 03:56:13,640 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 998e4efe5a1742a9998a6135dbaf037d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1682567771575 2023-04-27 03:56:13,640 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 29dca4d280b04757be7e16e7f1cedab9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1682567771666 2023-04-27 03:56:13,640 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting b07ea9d691bc45bb91a97c337c415045, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1682567771666 2023-04-27 03:56:13,640 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 23fc139379e54279a94ce7664e467a1e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1682567772766 2023-04-27 03:56:13,640 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 7dd616c47e23491597947e082e84eb6e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1682567772766 2023-04-27 03:56:13,647 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#B#compaction#577 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:13,663 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#A#compaction#578 average throughput is 1.16 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:13,736 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:13,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:13,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:13,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:13,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:13,740 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:13,740 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:13,740 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:13,740 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:13,741 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,741 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,742 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:13,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:13,743 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,743 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,743 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:13,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:13,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:13,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:13,743 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:13,744 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:13,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:13,744 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:13,744 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:13,744 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:13,744 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:13,744 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:13,744 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:13,745 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:13,745 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-27 03:56:13,745 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:13,745 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:13,745 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:13,745 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:13,745 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:13,745 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:13,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:13,923 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:13,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:13,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567833949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,958 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:13,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567833955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:13,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567833958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:13,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567833968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:13,973 DEBUG [Thread-4047] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., hostname=jenkins-hbase4.apache.org,40885,1682567618305, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 03:56:14,056 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:14,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567834054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:14,065 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/fde2c83b615c4282aad592d98efe8932 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/fde2c83b615c4282aad592d98efe8932 2023-04-27 03:56:14,069 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/B of 1c25ca7c0495563171ae7ccdf0350231 into fde2c83b615c4282aad592d98efe8932(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:14,070 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:14,070 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/B, priority=13, startTime=1682567773637; duration=0sec 2023-04-27 03:56:14,070 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:14,070 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:56:14,071 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:56:14,071 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/C is initiating minor compaction (all files) 2023-04-27 03:56:14,071 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/C in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:14,071 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a34a4d1591b5423d99d50884dd85855e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/9306df8ba8af40139873762d875c48d0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/ad73db38881f4483aec0437c9cca4d6a] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=36.3 K 2023-04-27 03:56:14,071 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting a34a4d1591b5423d99d50884dd85855e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1682567771575 2023-04-27 03:56:14,072 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 9306df8ba8af40139873762d875c48d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1682567771666 2023-04-27 03:56:14,072 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting ad73db38881f4483aec0437c9cca4d6a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1682567772766 2023-04-27 03:56:14,077 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#C#compaction#580 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:14,084 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/d9c96c7509e34e40b7d43f6bcf30d8f0 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/d9c96c7509e34e40b7d43f6bcf30d8f0 2023-04-27 03:56:14,088 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/C of 1c25ca7c0495563171ae7ccdf0350231 into d9c96c7509e34e40b7d43f6bcf30d8f0(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:14,088 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:14,088 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/C, priority=13, startTime=1682567773637; duration=0sec 2023-04-27 03:56:14,088 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:14,121 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/0699339f9efa4259bcd99cd6e856dd2f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/0699339f9efa4259bcd99cd6e856dd2f 2023-04-27 03:56:14,125 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/A of 1c25ca7c0495563171ae7ccdf0350231 into 0699339f9efa4259bcd99cd6e856dd2f(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:14,125 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:14,125 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/A, priority=13, startTime=1682567773637; duration=0sec 2023-04-27 03:56:14,125 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:14,163 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304277a24a946976146ee9259062680d87aee to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304277a24a946976146ee9259062680d87aee 2023-04-27 03:56:14,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:14,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567834257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:14,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:14,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567834560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:14,568 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=393, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/5273011fed834436ab8c27207725b7cb 2023-04-27 03:56:14,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:14,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567834960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:14,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:14,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567834970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:14,990 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/c165aef8878741f69e64bf2f98f7acb4 2023-04-27 03:56:15,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:15,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567835062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:15,401 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/d4b9316ee7c64898bddab4ada5e14ef3 2023-04-27 03:56:15,404 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/5273011fed834436ab8c27207725b7cb as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/5273011fed834436ab8c27207725b7cb 2023-04-27 03:56:15,408 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/5273011fed834436ab8c27207725b7cb, entries=150, sequenceid=393, filesize=25.4 K 2023-04-27 03:56:15,408 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/c165aef8878741f69e64bf2f98f7acb4 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/c165aef8878741f69e64bf2f98f7acb4 2023-04-27 03:56:15,411 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/c165aef8878741f69e64bf2f98f7acb4, entries=150, sequenceid=393, filesize=11.9 K 2023-04-27 03:56:15,412 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/d4b9316ee7c64898bddab4ada5e14ef3 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/d4b9316ee7c64898bddab4ada5e14ef3 2023-04-27 03:56:15,415 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/d4b9316ee7c64898bddab4ada5e14ef3, entries=150, sequenceid=393, filesize=11.9 K 2023-04-27 03:56:15,416 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 1c25ca7c0495563171ae7ccdf0350231 in 1671ms, sequenceid=393, compaction requested=false 2023-04-27 03:56:15,416 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:15,416 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:15,416 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:15,416 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:15,416 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:15,416 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:15,416 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:15,418 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:15,418 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:15,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:15,419 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:15,419 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:15,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:15,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:15,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:15,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3606a69c[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:15,421 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:15,421 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:15,422 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:15,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:15,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:15,422 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:15,422 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:15,422 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:15,422 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:15,422 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:15,423 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:15,423 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:15,423 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:15,423 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:15,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:15,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:15,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:15,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:15,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:15,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:15,426 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:15,426 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:15,426 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:15,426 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:15,426 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:15,428 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:15,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:15,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:15,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:15,428 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:15,428 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:15,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:15,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:15,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:15,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:15,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:15,429 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:15,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:15,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:15,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:15,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:15,430 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:15,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:15,438 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:15,438 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:15,438 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:15,438 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:15,439 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:15,439 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:15,439 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:15,439 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:15,439 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:15,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:15,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:15,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:15,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:15,439 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:15,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:15,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:15,439 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:15,439 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:15,539 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:15,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:15,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:15,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:15,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:15,543 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:15,543 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:15,543 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:15,543 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:15,543 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:15,543 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:15,544 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:15,544 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:15,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:15,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:15,545 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:15,545 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:15,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:15,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:15,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:15,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:15,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:15,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:15,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:15,546 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:15,546 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:15,546 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:15,546 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:15,546 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:15,546 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:15,546 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:15,546 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-27 03:56:15,547 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:15,547 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:15,547 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:15,547 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:15,547 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:15,547 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:15,961 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042751dbcfa392424adb9bcaa9a59f8fe9d6 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042751dbcfa392424adb9bcaa9a59f8fe9d6 2023-04-27 03:56:15,966 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=407, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/e882a5288dd6476686eada4ca2f60ab4 2023-04-27 03:56:15,974 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/65beffef189346548550bb244840af9a 2023-04-27 03:56:15,982 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/0a37f25ae60e4a2cac772f7c5de56987 2023-04-27 03:56:15,986 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/e882a5288dd6476686eada4ca2f60ab4 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/e882a5288dd6476686eada4ca2f60ab4 2023-04-27 03:56:15,989 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/e882a5288dd6476686eada4ca2f60ab4, entries=150, sequenceid=407, filesize=25.4 K 2023-04-27 03:56:15,990 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/65beffef189346548550bb244840af9a as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/65beffef189346548550bb244840af9a 2023-04-27 03:56:15,993 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/65beffef189346548550bb244840af9a, entries=150, sequenceid=407, filesize=11.9 K 2023-04-27 03:56:15,993 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/0a37f25ae60e4a2cac772f7c5de56987 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/0a37f25ae60e4a2cac772f7c5de56987 2023-04-27 03:56:15,997 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/0a37f25ae60e4a2cac772f7c5de56987, entries=150, sequenceid=407, filesize=11.9 K 2023-04-27 03:56:15,997 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 1c25ca7c0495563171ae7ccdf0350231 in 451ms, sequenceid=407, compaction requested=true 2023-04-27 03:56:15,998 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:15,998 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:15,998 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:15,998 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:15,998 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:15,998 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:15,998 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:15,999 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:15,999 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:15,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:16,000 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:16,000 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:16,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:16,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:16,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:16,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d68c3cb[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:16,001 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:16,001 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:16,002 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:16,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:16,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:16,003 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:16,003 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,003 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:16,003 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:16,003 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:16,005 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:16,005 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:16,005 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,005 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:16,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:16,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:16,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:16,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:16,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:16,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:16,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:16,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:16,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:16,008 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:16,008 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:16,008 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:16,008 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:16,008 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:16,009 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,009 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:16,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:16,009 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:16,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:16,010 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:16,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:16,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:16,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:16,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:16,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,011 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:16,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:16,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,016 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,016 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:16,016 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:16,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:16,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:16,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:16,016 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:16,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:16,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:16,016 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:16,016 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:16,017 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,017 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:16,017 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:16,017 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,017 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:16,017 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:16,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:16,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:16,117 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:16,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:16,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:16,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:16,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:16,120 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:16,120 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:16,120 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:16,120 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:16,121 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,121 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:16,123 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,123 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:16,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:16,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:16,123 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,123 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:16,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:16,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:16,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:16,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:16,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:16,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:16,126 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:16,126 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:16,126 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:16,126 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:16,127 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:16,127 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:16,127 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:16,127 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-27 03:56:16,127 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:16,127 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:16,127 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:16,127 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:16,127 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:16,127 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:16,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:16,131 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:16,139 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042745e691e0e55d4b56b855aada087661ac to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042745e691e0e55d4b56b855aada087661ac 2023-04-27 03:56:16,144 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=416, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/1ad567a9e4b849c5b939ae8f061bf871 2023-04-27 03:56:16,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567776177, max timestamp=1682567776190Num uniques 859; 2023-04-27 03:56:16,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567776177, max timestamp=1682567776190Num uniques 859; 2023-04-27 03:56:16,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:56:16,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567776177, max timestamp=1682567776190Num uniques 859; 2023-04-27 03:56:16,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:56:16,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:56:16,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:16,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567836239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:16,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567836342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,548 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:16,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567836546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,553 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/cdae90ce064c4870b89bba23388aa807 2023-04-27 03:56:16,565 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/11040a64a432490ea7a053b589445191 2023-04-27 03:56:16,569 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/1ad567a9e4b849c5b939ae8f061bf871 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/1ad567a9e4b849c5b939ae8f061bf871 2023-04-27 03:56:16,573 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/1ad567a9e4b849c5b939ae8f061bf871, entries=100, sequenceid=416, filesize=18.6 K 2023-04-27 03:56:16,574 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/cdae90ce064c4870b89bba23388aa807 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/cdae90ce064c4870b89bba23388aa807 2023-04-27 03:56:16,577 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/cdae90ce064c4870b89bba23388aa807, entries=100, sequenceid=416, filesize=9.5 K 2023-04-27 03:56:16,578 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/11040a64a432490ea7a053b589445191 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/11040a64a432490ea7a053b589445191 2023-04-27 03:56:16,581 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/11040a64a432490ea7a053b589445191, entries=50, sequenceid=416, filesize=7.1 K 2023-04-27 03:56:16,582 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 1c25ca7c0495563171ae7ccdf0350231 in 455ms, sequenceid=416, compaction requested=true 2023-04-27 03:56:16,582 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:16,582 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:16,582 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:16,582 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:16,582 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,582 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:16,582 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:16,584 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:16,584 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:16,584 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:16,584 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:16,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:16,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:16,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:16,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:16,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18df3b37[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:16,586 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:16,586 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:16,587 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:16,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:16,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:16,587 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:16,587 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:16,587 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,587 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:16,587 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:16,588 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,588 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:16,589 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:16,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:16,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:16,589 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:16,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:16,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:16,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:16,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:16,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:16,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:16,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:16,591 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:16,591 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:16,591 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:16,591 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:16,591 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:16,594 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,594 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,594 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:16,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:16,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:16,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:16,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:16,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:16,595 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:16,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:16,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:16,596 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:16,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:16,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,598 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,598 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:16,598 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:16,598 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,598 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:16,598 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:16,598 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,598 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:16,598 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:16,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:16,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:16,598 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:16,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:16,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:16,599 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:16,599 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:16,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:16,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:16,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:16,699 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:16,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:16,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:16,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:16,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:16,702 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:16,702 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:16,703 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:16,703 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:16,703 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,703 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:16,704 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:16,704 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:16,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:16,704 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,705 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:16,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:16,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:16,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:16,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:16,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:16,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:16,705 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:16,705 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:16,705 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:16,706 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:16,706 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:16,706 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:16,706 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:16,706 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-27 03:56:16,706 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:16,706 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:16,706 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:16,706 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:16,707 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:16,707 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:16,724 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042733db045108874f65a19eda56ca9659c6 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042733db045108874f65a19eda56ca9659c6 2023-04-27 03:56:16,728 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=453, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/5034e204cc9d444ab0741ca8113d060e 2023-04-27 03:56:16,737 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/7de020e9a7f34c309f294667389c7241 2023-04-27 03:56:16,745 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/cd79675550354cf9bfb2148b5c5fb386 2023-04-27 03:56:16,749 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/5034e204cc9d444ab0741ca8113d060e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/5034e204cc9d444ab0741ca8113d060e 2023-04-27 03:56:16,754 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/5034e204cc9d444ab0741ca8113d060e, entries=150, sequenceid=453, filesize=25.4 K 2023-04-27 03:56:16,755 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/7de020e9a7f34c309f294667389c7241 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/7de020e9a7f34c309f294667389c7241 2023-04-27 03:56:16,758 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/7de020e9a7f34c309f294667389c7241, entries=150, sequenceid=453, filesize=11.9 K 2023-04-27 03:56:16,759 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/cd79675550354cf9bfb2148b5c5fb386 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/cd79675550354cf9bfb2148b5c5fb386 2023-04-27 03:56:16,762 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/cd79675550354cf9bfb2148b5c5fb386, entries=150, sequenceid=453, filesize=11.9 K 2023-04-27 03:56:16,763 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=0 B/0 for 1c25ca7c0495563171ae7ccdf0350231 in 57ms, sequenceid=453, compaction requested=true 2023-04-27 03:56:16,763 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:16,763 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:16,763 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:16,763 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:16,763 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,763 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:16,763 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:16,766 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,766 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:16,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:16,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:16,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:16,766 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:16,767 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:16,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:16,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33d8556a[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:16,768 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:16,768 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:16,770 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:16,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:16,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:16,771 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:16,771 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,771 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:16,771 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:16,771 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:16,772 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:16,772 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,772 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:16,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:16,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:16,772 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:16,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:16,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:16,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:16,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:16,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:16,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:16,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:16,775 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:16,775 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:16,775 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:16,775 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:16,775 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:16,777 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,777 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:16,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:16,777 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:16,777 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:16,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:16,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:16,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:16,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:16,778 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:16,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:16,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:16,780 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:16,781 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:16,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:16,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:16,781 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,781 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:16,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:16,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:16,781 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:16,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:16,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:16,781 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:16,781 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:16,781 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:16,781 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,781 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:16,781 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:16,797 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:16,881 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:16,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:16,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:16,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:16,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:16,885 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:16,885 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:16,885 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:16,885 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:16,885 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,885 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:16,887 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:16,887 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:16,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:16,887 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:16,887 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:16,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:16,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:16,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:16,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:16,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:16,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:16,889 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:16,889 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:16,889 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:16,889 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:16,889 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:16,889 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:16,889 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:16,889 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-27 03:56:16,889 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:16,889 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:16,889 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:16,889 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:16,889 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:16,889 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:16,905 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304275fdc8a4a8fb54b0e9e870f4b1e63b0f3 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304275fdc8a4a8fb54b0e9e870f4b1e63b0f3 2023-04-27 03:56:16,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:16,905 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:16,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567776958, max timestamp=1682567776968Num uniques 859; 2023-04-27 03:56:16,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567776958, max timestamp=1682567776968Num uniques 859; 2023-04-27 03:56:16,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:56:16,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567776958, max timestamp=1682567776968Num uniques 859; 2023-04-27 03:56:16,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:56:16,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:56:17,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682567777002, max timestamp=1682567777013Num uniques 1018; 2023-04-27 03:56:17,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682567777002, max timestamp=1682567777013Num uniques 1018; 2023-04-27 03:56:17,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682567777002, max timestamp=1682567777013Num uniques 1018; 2023-04-27 03:56:17,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-27 03:56:17,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-27 03:56:17,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-27 03:56:17,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8098, totalHeapSize=18648, min timestamp=1682567777026, max timestamp=1682567777042Num uniques 1077; 2023-04-27 03:56:17,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49318, new segment data size=8098, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-27 03:56:17,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8092, totalHeapSize=18648, min timestamp=1682567777026, max timestamp=1682567777042Num uniques 1077; 2023-04-27 03:56:17,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49312, new segment data size=8092, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-27 03:56:17,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8099, totalHeapSize=18648, min timestamp=1682567777026, max timestamp=1682567777042Num uniques 1077; 2023-04-27 03:56:17,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49319, new segment data size=8099, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-27 03:56:17,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682567777063, max timestamp=1682567777071Num uniques 1036; 2023-04-27 03:56:17,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-27 03:56:17,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682567777063, max timestamp=1682567777071Num uniques 1036; 2023-04-27 03:56:17,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-27 03:56:17,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682567777063, max timestamp=1682567777071Num uniques 1036; 2023-04-27 03:56:17,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-27 03:56:17,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682567777092, max timestamp=1682567777100Num uniques 1095; 2023-04-27 03:56:17,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-27 03:56:17,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682567777092, max timestamp=1682567777100Num uniques 1095; 2023-04-27 03:56:17,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682567777092, max timestamp=1682567777100Num uniques 1095; 2023-04-27 03:56:17,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-27 03:56:17,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-27 03:56:17,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682567777117, max timestamp=1682567777133Num uniques 1104; 2023-04-27 03:56:17,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-27 03:56:17,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682567777117, max timestamp=1682567777133Num uniques 1104; 2023-04-27 03:56:17,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-27 03:56:17,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682567777117, max timestamp=1682567777133Num uniques 1104; 2023-04-27 03:56:17,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-27 03:56:17,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682567777153, max timestamp=1682567777164Num uniques 1063; 2023-04-27 03:56:17,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682567777153, max timestamp=1682567777164Num uniques 1063; 2023-04-27 03:56:17,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-27 03:56:17,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-27 03:56:17,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682567777153, max timestamp=1682567777164Num uniques 1063; 2023-04-27 03:56:17,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-27 03:56:17,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682567777190, max timestamp=1682567777199Num uniques 1022; 2023-04-27 03:56:17,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682567777190, max timestamp=1682567777199Num uniques 1022; 2023-04-27 03:56:17,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-27 03:56:17,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-27 03:56:17,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682567777190, max timestamp=1682567777199Num uniques 1022; 2023-04-27 03:56:17,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-27 03:56:17,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682567777223, max timestamp=1682567777231Num uniques 1031; 2023-04-27 03:56:17,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-27 03:56:17,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682567777223, max timestamp=1682567777231Num uniques 1031; 2023-04-27 03:56:17,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-27 03:56:17,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682567777223, max timestamp=1682567777231Num uniques 1031; 2023-04-27 03:56:17,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-27 03:56:17,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682567777254, max timestamp=1682567777261Num uniques 1090; 2023-04-27 03:56:17,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-27 03:56:17,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682567777254, max timestamp=1682567777261Num uniques 1090; 2023-04-27 03:56:17,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-27 03:56:17,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682567777254, max timestamp=1682567777261Num uniques 1090; 2023-04-27 03:56:17,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-27 03:56:17,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1682567777272, max timestamp=1682567777286Num uniques 1049; 2023-04-27 03:56:17,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=13694, suffix heap size=137064,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=299 2023-04-27 03:56:17,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1682567777272, max timestamp=1682567777286Num uniques 1049; 2023-04-27 03:56:17,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=13694, suffix heap size=137064,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=299 2023-04-27 03:56:17,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682567777281, max timestamp=1682567777286Num uniques 1049; 2023-04-27 03:56:17,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-27 03:56:17,314 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=459, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/b74c4ee927fd47cdbe9df5b3fb42f695 2023-04-27 03:56:17,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682567777303, max timestamp=1682567777312Num uniques 1158; 2023-04-27 03:56:17,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682567777303, max timestamp=1682567777312Num uniques 1158; 2023-04-27 03:56:17,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682567777303, max timestamp=1682567777312Num uniques 1108; 2023-04-27 03:56:17,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=9520, suffix heap size=148400,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=208 2023-04-27 03:56:17,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=9520, suffix heap size=148400,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=208 2023-04-27 03:56:17,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-27 03:56:17,323 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/9006d5cdeb1a4aebae292e9809884531 2023-04-27 03:56:17,331 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/33f2ac778d57433e841795abc5e94ce2 2023-04-27 03:56:17,335 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/b74c4ee927fd47cdbe9df5b3fb42f695 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/b74c4ee927fd47cdbe9df5b3fb42f695 2023-04-27 03:56:17,338 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/b74c4ee927fd47cdbe9df5b3fb42f695, entries=150, sequenceid=459, filesize=25.4 K 2023-04-27 03:56:17,338 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/9006d5cdeb1a4aebae292e9809884531 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/9006d5cdeb1a4aebae292e9809884531 2023-04-27 03:56:17,341 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/9006d5cdeb1a4aebae292e9809884531, entries=100, sequenceid=459, filesize=9.5 K 2023-04-27 03:56:17,341 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/33f2ac778d57433e841795abc5e94ce2 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/33f2ac778d57433e841795abc5e94ce2 2023-04-27 03:56:17,344 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/33f2ac778d57433e841795abc5e94ce2, entries=100, sequenceid=459, filesize=9.5 K 2023-04-27 03:56:17,345 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=120.76 KB/123660 for 1c25ca7c0495563171ae7ccdf0350231 in 456ms, sequenceid=459, compaction requested=true 2023-04-27 03:56:17,345 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:17,345 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:17,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:17,345 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=127.47 KB heapSize=360.47 KB 2023-04-27 03:56:17,345 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:17,345 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:17,345 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,345 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:17,345 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:17,345 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:17,345 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:17,345 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:17,345 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:17,345 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:17,346 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:17,347 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:17,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:17,347 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:17,347 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,348 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:17,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:17,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:17,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:17,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@306927cb[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:17,349 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:17,349 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,350 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,350 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:17,350 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:17,350 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:17,351 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,351 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:17,352 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:17,352 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:17,352 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:17,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:17,352 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:17,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:17,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:17,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:17,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:17,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:17,356 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:17,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:17,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:17,356 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:17,356 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:17,357 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304270f0087ef994a4ba8a69bd4135ebec207 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304270f0087ef994a4ba8a69bd4135ebec207 2023-04-27 03:56:17,359 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,359 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:17,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:17,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,359 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:17,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:17,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:17,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:17,360 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:17,362 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,362 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=684, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/aebc5e7cc2e64842a582a5c07ec7d2e5 2023-04-27 03:56:17,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:17,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,364 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,364 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:17,364 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,364 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:17,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:17,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:17,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:17,364 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:17,364 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:17,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:17,364 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,365 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,365 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,365 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,365 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:17,365 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:17,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:17,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:17,366 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:17,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567837359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:17,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567837361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,372 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:17,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567837367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,465 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:17,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:17,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:17,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:17,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:17,469 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:17,469 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:17,469 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:17,469 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:17,469 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,469 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,470 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:17,470 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:17,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:17,471 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,471 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:17,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:17,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:17,471 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:17,471 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:17,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:17,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:17,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:17,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:17,472 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:17,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567837468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:17,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:17,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:17,472 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:17,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567837469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,472 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:17,472 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:17,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:17,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:17,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:17,472 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:17,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:17,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567837473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,478 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:17,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:17,478 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,478 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,478 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:17,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:17,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:17,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:17,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:17,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bc4dc96[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:17,479 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:17,479 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,480 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:17,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:17,481 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:17,481 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,481 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:17,482 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,482 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:17,482 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:17,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:17,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:17,482 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:17,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:17,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:17,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:17,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:17,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:17,485 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:17,485 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:17,485 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:17,485 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:17,485 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:17,486 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,486 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:17,486 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:17,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:17,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:17,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:17,487 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:17,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:17,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,488 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:17,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,491 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,491 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,491 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:17,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:17,491 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:17,491 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:17,491 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:17,491 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:17,491 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:17,491 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:17,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:17,491 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,491 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:17,491 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:17,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:17,592 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:17,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:17,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:17,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:17,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:17,595 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:17,595 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:17,595 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:17,595 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:17,596 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,596 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,597 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,597 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:17,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:17,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:17,597 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,597 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:17,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:17,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:17,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:17,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:17,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:17,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:17,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:17,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:17,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:17,604 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:17,604 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:17,604 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:17,604 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:17,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:17,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:17,604 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,605 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:17,605 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:17,606 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:17,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:17,606 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,607 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:17,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:17,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:17,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:17,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:17,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a23545a[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:17,608 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:17,608 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,609 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,609 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:17,609 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:17,609 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:17,609 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,609 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:17,611 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:17,611 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:17,611 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,611 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:17,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:17,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:17,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:17,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:17,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:17,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:17,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:17,613 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:17,613 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:17,613 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:17,613 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:17,613 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:17,615 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,615 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,615 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:17,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:17,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:17,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:17,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:17,615 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:17,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:17,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,616 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:17,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,619 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,619 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,619 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,619 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:17,619 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:17,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:17,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:17,619 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,619 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,619 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,619 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,619 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:17,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:17,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:17,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:17,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:17,620 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:17,620 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:17,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:17,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567837673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:17,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567837674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,681 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:17,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567837676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,720 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:17,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:17,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:17,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:17,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:17,725 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:17,725 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:17,725 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:17,725 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:17,726 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,726 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,727 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:17,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:17,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:17,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,727 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:17,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:17,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,728 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:17,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:17,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:17,728 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:17,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:17,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:17,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:17,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:17,729 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:17,729 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:17,729 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:17,729 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:17,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:17,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:17,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:17,729 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:17,732 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:17,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:17,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,732 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:17,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:17,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:17,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ba13371[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:17,734 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:17,734 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,735 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,735 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:17,735 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,735 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:17,735 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:17,735 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:17,737 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:17,737 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:17,737 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,737 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:17,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:17,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:17,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:17,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:17,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:17,740 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:17,740 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:17,740 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:17,740 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:17,740 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:17,741 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,741 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,741 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:17,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:17,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:17,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:17,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:17,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:17,742 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:17,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:17,744 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,746 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,746 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:17,747 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,747 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:17,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:17,747 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:17,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:17,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:17,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:17,747 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,747 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,747 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,747 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,747 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,747 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:17,747 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:17,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:17,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:17,773 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=684 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/dc241d561b19441e9405a1086a449584 2023-04-27 03:56:17,847 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:17,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:17,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:17,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:17,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:17,850 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:17,850 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:17,851 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:17,851 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:17,851 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,851 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,852 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:17,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:17,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:17,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,852 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:17,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:17,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,853 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:17,853 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:17,853 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:17,853 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:17,853 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:17,853 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:17,854 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:17,854 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:17,854 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:17,854 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:17,854 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:17,854 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:17,854 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:17,854 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:17,854 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,854 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:17,854 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:17,855 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,855 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:17,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:17,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,856 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:17,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:17,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:17,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:17,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:17,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20f14b1[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:17,857 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:17,857 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,862 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,862 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,863 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:17,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:17,863 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:17,863 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:17,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,864 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:17,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:17,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:17,864 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:17,864 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:17,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:17,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:17,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:17,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:17,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:17,866 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:17,866 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:17,866 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:17,866 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:17,866 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:17,868 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,868 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,868 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:17,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:17,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:17,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:17,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:17,868 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:17,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:17,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,870 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:17,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,872 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,872 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,872 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:17,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:17,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:17,872 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:17,872 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:17,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:17,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:17,872 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,872 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:17,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:17,872 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:17,872 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,873 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,873 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,873 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:17,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:17,973 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:17,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:17,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:17,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:17,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:17,976 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:17,976 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:17,976 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:17,976 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:17,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,977 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:17,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567837977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,980 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,980 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:17,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:17,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:17,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,981 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,981 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:17,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:17,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567837978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:17,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:17,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:17,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:17,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:17,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:17,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:17,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:17,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:17,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:17,982 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:17,982 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:17,982 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:17,982 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:17,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:17,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:17,982 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:17,983 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:17,984 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,984 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:17,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:17,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:17,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:17,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567837982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:17,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:17,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:17,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:17,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f62480a[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:17,986 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:17,986 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,987 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:17,987 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:17,987 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:17,988 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:17,988 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,988 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:17,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:17,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:17,989 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:17,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:17,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:17,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:17,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:17,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:17,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:17,993 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:17,993 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:17,993 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:17,993 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:17,993 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:17,995 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,995 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,995 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:17,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:17,996 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:17,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,998 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:17,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:17,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:17,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,000 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,000 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:18,000 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:18,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:18,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:18,000 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:18,000 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:18,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:18,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:18,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:18,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:18,000 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:18,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48444 deadline: 1682567837999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,001 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:18,001 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:18,001 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,001 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:18,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:18,001 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,001 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,002 DEBUG [Thread-4047] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8171 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., hostname=jenkins-hbase4.apache.org,40885,1682567618305, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 03:56:18,101 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:18,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:18,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:18,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:18,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:18,104 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:18,104 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:18,104 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:18,104 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:18,105 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,105 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:18,106 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:18,106 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:18,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:18,107 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,107 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:18,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:18,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:18,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:18,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:18,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:18,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:18,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:18,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:18,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:18,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:18,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:18,108 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:18,108 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:18,108 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:18,108 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:18,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:18,108 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:18,109 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,109 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:18,109 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:18,110 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,110 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:18,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:18,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:18,111 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,111 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:18,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:18,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:18,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:18,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:18,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f735438[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:18,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:18,113 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,115 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,115 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:18,115 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:18,115 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,115 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:18,115 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:18,116 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,116 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:18,116 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:18,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:18,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:18,116 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:18,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:18,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:18,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:18,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:18,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:18,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:18,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:18,120 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:18,120 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:18,120 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:18,120 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:18,120 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:18,121 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,121 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,121 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:18,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:18,122 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:18,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:18,123 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:18,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:18,125 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,125 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:18,125 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:18,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:18,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:18,125 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:18,125 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:18,126 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,126 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,126 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,126 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:18,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:18,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:18,125 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:18,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:18,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:18,126 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:18,126 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:18,183 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=684 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/59a1ac71baad47cf845edb6c269a2cba 2023-04-27 03:56:18,186 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/aebc5e7cc2e64842a582a5c07ec7d2e5 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/aebc5e7cc2e64842a582a5c07ec7d2e5 2023-04-27 03:56:18,190 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/aebc5e7cc2e64842a582a5c07ec7d2e5, entries=250, sequenceid=684, filesize=39.2 K 2023-04-27 03:56:18,190 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/dc241d561b19441e9405a1086a449584 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/dc241d561b19441e9405a1086a449584 2023-04-27 03:56:18,194 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/dc241d561b19441e9405a1086a449584, entries=150, sequenceid=684, filesize=11.9 K 2023-04-27 03:56:18,194 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/59a1ac71baad47cf845edb6c269a2cba as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/59a1ac71baad47cf845edb6c269a2cba 2023-04-27 03:56:18,197 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/59a1ac71baad47cf845edb6c269a2cba, entries=150, sequenceid=684, filesize=11.9 K 2023-04-27 03:56:18,198 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.34 KB/389472, currentSize=46.96 KB/48090 for 1c25ca7c0495563171ae7ccdf0350231 in 853ms, sequenceid=684, compaction requested=true 2023-04-27 03:56:18,198 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:18,198 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:18,198 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-27 03:56:18,198 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-27 03:56:18,198 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:18,198 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:18,200 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 190068 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-27 03:56:18,200 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 81128 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-27 03:56:18,201 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/A is initiating minor compaction (all files) 2023-04-27 03:56:18,201 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/B is initiating minor compaction (all files) 2023-04-27 03:56:18,201 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/A in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:18,201 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/B in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:18,201 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/0699339f9efa4259bcd99cd6e856dd2f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/5273011fed834436ab8c27207725b7cb, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/e882a5288dd6476686eada4ca2f60ab4, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/1ad567a9e4b849c5b939ae8f061bf871, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/5034e204cc9d444ab0741ca8113d060e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/b74c4ee927fd47cdbe9df5b3fb42f695, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/aebc5e7cc2e64842a582a5c07ec7d2e5] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=185.6 K 2023-04-27 03:56:18,201 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/fde2c83b615c4282aad592d98efe8932, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/c165aef8878741f69e64bf2f98f7acb4, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/65beffef189346548550bb244840af9a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/cdae90ce064c4870b89bba23388aa807, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/7de020e9a7f34c309f294667389c7241, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/9006d5cdeb1a4aebae292e9809884531, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/dc241d561b19441e9405a1086a449584] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=79.2 K 2023-04-27 03:56:18,201 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting fde2c83b615c4282aad592d98efe8932, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1682567772766 2023-04-27 03:56:18,201 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 0699339f9efa4259bcd99cd6e856dd2f, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1682567772766 2023-04-27 03:56:18,202 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 5273011fed834436ab8c27207725b7cb, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1682567772824 2023-04-27 03:56:18,202 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting c165aef8878741f69e64bf2f98f7acb4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1682567772824 2023-04-27 03:56:18,203 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 65beffef189346548550bb244840af9a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1682567773938 2023-04-27 03:56:18,203 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting e882a5288dd6476686eada4ca2f60ab4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1682567773938 2023-04-27 03:56:18,203 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting cdae90ce064c4870b89bba23388aa807, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1682567776109 2023-04-27 03:56:18,203 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 1ad567a9e4b849c5b939ae8f061bf871, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1682567776109 2023-04-27 03:56:18,203 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 7de020e9a7f34c309f294667389c7241, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1682567776226 2023-04-27 03:56:18,203 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 5034e204cc9d444ab0741ca8113d060e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1682567776226 2023-04-27 03:56:18,204 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 9006d5cdeb1a4aebae292e9809884531, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1682567776866 2023-04-27 03:56:18,204 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting b74c4ee927fd47cdbe9df5b3fb42f695, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1682567776866 2023-04-27 03:56:18,204 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting dc241d561b19441e9405a1086a449584, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=684, earliestPutTs=1682567777335 2023-04-27 03:56:18,204 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting aebc5e7cc2e64842a582a5c07ec7d2e5, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=684, earliestPutTs=1682567777335 2023-04-27 03:56:18,214 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#B#compaction#599 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:18,217 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#A#compaction#598 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:18,226 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:18,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:18,229 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/c6f77fb963ca44e182608848d1180758 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/c6f77fb963ca44e182608848d1180758 2023-04-27 03:56:18,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:18,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:18,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:18,232 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:18,232 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:18,232 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:18,232 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:18,233 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,233 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:18,236 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:18,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:18,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:18,236 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:18,237 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,237 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:18,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:18,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:18,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:18,238 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:18,238 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:18,238 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:18,238 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:18,238 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:18,238 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:18,238 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:18,239 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:18,239 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:18,239 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-27 03:56:18,239 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:18,239 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:18,239 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:18,239 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:18,239 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:18,239 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:18,241 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/B of 1c25ca7c0495563171ae7ccdf0350231 into c6f77fb963ca44e182608848d1180758(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:18,241 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:18,241 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/B, priority=9, startTime=1682567778198; duration=0sec 2023-04-27 03:56:18,241 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:18,241 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-27 03:56:18,243 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 78654 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-27 03:56:18,243 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/C is initiating minor compaction (all files) 2023-04-27 03:56:18,243 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/C in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:18,243 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/d9c96c7509e34e40b7d43f6bcf30d8f0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/d4b9316ee7c64898bddab4ada5e14ef3, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/0a37f25ae60e4a2cac772f7c5de56987, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/11040a64a432490ea7a053b589445191, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/cd79675550354cf9bfb2148b5c5fb386, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/33f2ac778d57433e841795abc5e94ce2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/59a1ac71baad47cf845edb6c269a2cba] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=76.8 K 2023-04-27 03:56:18,244 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting d9c96c7509e34e40b7d43f6bcf30d8f0, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1682567772766 2023-04-27 03:56:18,244 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting d4b9316ee7c64898bddab4ada5e14ef3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1682567772824 2023-04-27 03:56:18,244 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 0a37f25ae60e4a2cac772f7c5de56987, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1682567773938 2023-04-27 03:56:18,245 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 11040a64a432490ea7a053b589445191, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1682567776124 2023-04-27 03:56:18,245 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting cd79675550354cf9bfb2148b5c5fb386, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1682567776226 2023-04-27 03:56:18,246 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 33f2ac778d57433e841795abc5e94ce2, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1682567776866 2023-04-27 03:56:18,246 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 59a1ac71baad47cf845edb6c269a2cba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=684, earliestPutTs=1682567777335 2023-04-27 03:56:18,251 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/c47f0f651ec9463c922f9fc107748a13 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/c47f0f651ec9463c922f9fc107748a13 2023-04-27 03:56:18,256 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/A of 1c25ca7c0495563171ae7ccdf0350231 into c47f0f651ec9463c922f9fc107748a13(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:18,256 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:18,256 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/A, priority=9, startTime=1682567778198; duration=0sec 2023-04-27 03:56:18,256 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:18,258 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#C#compaction#601 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:18,265 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304272d7256610b8447dc8e4b6765aa58203f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272d7256610b8447dc8e4b6765aa58203f 2023-04-27 03:56:18,275 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/042d4cd9393d424fae7a24be13462db3 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/042d4cd9393d424fae7a24be13462db3 2023-04-27 03:56:18,279 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/C of 1c25ca7c0495563171ae7ccdf0350231 into 042d4cd9393d424fae7a24be13462db3(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:18,279 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:18,279 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/C, priority=9, startTime=1682567778198; duration=0sec 2023-04-27 03:56:18,279 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:18,281 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=694, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/7e149abbc3d442c893cedd5e7bb51d57 2023-04-27 03:56:18,305 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/59f63a26c64b4bbfbdf339ae5f4620f8 2023-04-27 03:56:18,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:18,482 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:18,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567778498, max timestamp=1682567778507Num uniques 859; 2023-04-27 03:56:18,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:56:18,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567778498, max timestamp=1682567778507Num uniques 859; 2023-04-27 03:56:18,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:56:18,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567778498, max timestamp=1682567778507Num uniques 859; 2023-04-27 03:56:18,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:56:18,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:18,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:18,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567838524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567838525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:18,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567838528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:18,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:18,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567838629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567838629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:18,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567838632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,716 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/1aa21016a1b64319b4275be3de59f242 2023-04-27 03:56:18,720 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/7e149abbc3d442c893cedd5e7bb51d57 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/7e149abbc3d442c893cedd5e7bb51d57 2023-04-27 03:56:18,723 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/7e149abbc3d442c893cedd5e7bb51d57, entries=100, sequenceid=694, filesize=18.6 K 2023-04-27 03:56:18,723 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/59f63a26c64b4bbfbdf339ae5f4620f8 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/59f63a26c64b4bbfbdf339ae5f4620f8 2023-04-27 03:56:18,726 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/59f63a26c64b4bbfbdf339ae5f4620f8, entries=100, sequenceid=694, filesize=9.5 K 2023-04-27 03:56:18,727 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/1aa21016a1b64319b4275be3de59f242 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/1aa21016a1b64319b4275be3de59f242 2023-04-27 03:56:18,730 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/1aa21016a1b64319b4275be3de59f242, entries=100, sequenceid=694, filesize=9.5 K 2023-04-27 03:56:18,731 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1c25ca7c0495563171ae7ccdf0350231 in 492ms, sequenceid=694, compaction requested=false 2023-04-27 03:56:18,731 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:18,731 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:18,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:18,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:18,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:18,731 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:18,733 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,733 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:18,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:18,733 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:18,733 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:18,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:18,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:18,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:18,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:18,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d3bde1c[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:18,734 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:18,734 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,735 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,736 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:18,736 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:18,736 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:18,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,736 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:18,741 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:18,741 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:18,741 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,741 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:18,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:18,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:18,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:18,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:18,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:18,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:18,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:18,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:18,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:18,744 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:18,745 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:18,745 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:18,745 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:18,745 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:18,746 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,746 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,746 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:18,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:18,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:18,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:18,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:18,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:18,747 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:18,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:18,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:18,748 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:18,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:18,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:18,751 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,751 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:18,751 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:18,751 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,751 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,751 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,751 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,751 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:18,751 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:18,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:18,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:18,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:18,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:18,751 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:18,751 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:18,752 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:18,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:18,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:18,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:18,836 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-27 03:56:18,836 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:18,836 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:18,836 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:18,836 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:18,836 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:18,836 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:18,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:18,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567838843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:18,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567838843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:18,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567838844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,852 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:18,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:18,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:18,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:18,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:18,855 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:18,855 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:18,855 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:18,855 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:18,855 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,855 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:18,856 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,856 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:18,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:18,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:18,857 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,857 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:18,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:18,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:18,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:18,858 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:18,858 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:18,858 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:18,858 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:18,859 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:18,859 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:18,859 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:18,859 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:18,859 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:18,859 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:18,859 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:18,859 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:18,859 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:18,859 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:18,859 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,859 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:18,859 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:18,861 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,861 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:18,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,862 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:18,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:18,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:18,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@576bed96[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:18,863 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:18,863 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,864 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,865 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,865 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:18,865 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:18,865 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:18,865 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:18,866 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:18,866 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:18,867 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:18,867 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:18,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:18,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:18,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:18,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:18,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:18,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:18,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:18,870 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:18,870 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:18,870 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:18,870 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:18,870 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:18,871 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,872 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,872 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:18,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:18,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:18,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:18,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:18,872 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:18,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:18,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:18,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:18,873 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:18,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:18,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,877 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,877 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:18,877 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:18,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:18,877 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:18,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:18,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:18,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:18,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:18,877 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:18,877 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,878 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,877 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:18,878 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:18,878 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,878 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,878 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:18,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:18,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:18,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:18,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567838947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:18,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 371 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567838948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:18,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567838948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,978 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:18,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:18,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:18,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:18,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:18,981 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:18,981 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:18,981 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:18,981 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:18,982 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,982 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:18,983 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,983 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:18,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:18,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:18,983 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,983 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:18,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:18,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:18,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:18,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:18,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:18,984 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:18,984 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:18,984 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:18,984 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:18,984 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:18,985 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:18,985 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:18,985 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:18,985 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:18,985 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:18,986 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,986 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:18,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:18,987 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,987 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:18,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:18,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a94cd69[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:18,988 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:18,988 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,989 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:18,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:18,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:18,989 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:18,989 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,989 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:18,992 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:18,992 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,992 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:18,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:18,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:18,992 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:18,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:18,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:18,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:18,994 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:18,994 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:18,994 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:18,994 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:18,994 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:18,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:18,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:18,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,996 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:18,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:18,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:18,996 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:18,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:18,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:18,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,000 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:19,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:19,006 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:19,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:19,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:19,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:19,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,006 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,007 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,007 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:19,007 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:19,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:19,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:19,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:19,107 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:19,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:19,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:19,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:19,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:19,110 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:19,110 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:19,110 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:19,110 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:19,111 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,111 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,112 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:19,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:19,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:19,112 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,112 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,112 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:19,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:19,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:19,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:19,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:19,113 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:19,113 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:19,113 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:19,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:19,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:19,114 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:19,114 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:19,114 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:19,114 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:19,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:19,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:19,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:19,114 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:19,117 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,117 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:19,117 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,117 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:19,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:19,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f838a97[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:19,119 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:19,119 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,120 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:19,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:19,121 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:19,121 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,121 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:19,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:19,122 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:19,122 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:19,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:19,125 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:19,125 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:19,125 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:19,125 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:19,125 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:19,126 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,126 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:19,126 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:19,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,128 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:19,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,129 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,135 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,135 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,135 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,135 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,135 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,135 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,135 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,135 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:19,135 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:19,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:19,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:19,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:19,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:19,135 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:19,136 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:19,136 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:19,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:19,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:19,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:19,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567839151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:19,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567839151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:19,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567839152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,236 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:19,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:19,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:19,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:19,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:19,238 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:19,238 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:19,238 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:19,238 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:19,239 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,239 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,240 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:19,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:19,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:19,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,240 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:19,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:19,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:19,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:19,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:19,241 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:19,241 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:19,241 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:19,241 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:19,241 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:19,241 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:19,241 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:19,242 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:19,242 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:19,242 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:19,242 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:19,242 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,242 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:19,242 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:19,243 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:19,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,243 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:19,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:19,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cbb6645[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:19,245 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:19,245 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,246 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,246 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:19,246 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:19,246 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:19,247 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,247 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:19,248 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:19,248 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:19,248 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:19,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:19,248 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:19,248 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427bc911190f2824a3ca5c2b6c753a2be87 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427bc911190f2824a3ca5c2b6c753a2be87 2023-04-27 03:56:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:19,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:19,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:19,251 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:19,251 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:19,251 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:19,251 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:19,251 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:19,258 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=734, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/a91e43513d9741c191808d3201eddfbe 2023-04-27 03:56:19,258 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,258 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,259 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:19,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:19,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:19,260 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:19,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,261 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:19,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,264 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:19,264 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:19,264 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,264 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,264 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,264 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,264 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,264 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,264 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,264 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:19,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:19,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:19,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:19,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:19,265 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:19,265 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:19,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:19,365 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:19,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:19,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:19,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:19,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:19,368 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:19,368 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:19,369 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:19,369 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:19,369 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,369 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,370 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,370 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:19,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:19,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:19,371 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,371 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:19,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:19,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:19,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:19,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:19,372 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:19,372 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:19,372 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:19,372 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:19,372 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:19,372 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:19,373 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:19,373 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:19,373 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:19,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:19,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:19,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:19,373 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:19,375 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,375 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:19,375 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:19,375 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:19,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:19,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:19,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34343b19[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:19,376 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:19,377 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,378 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,378 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:19,378 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:19,378 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:19,378 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,378 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:19,379 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,379 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:19,379 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:19,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:19,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:19,379 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:19,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:19,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:19,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:19,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:19,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:19,382 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:19,382 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:19,382 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:19,382 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:19,382 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:19,384 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,384 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:19,384 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:19,384 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,385 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:19,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,388 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,388 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,389 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:19,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:19,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:19,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:19,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:19,389 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:19,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:19,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:19,389 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:19,389 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:19,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:19,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:19,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:19,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567839453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:19,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 375 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567839454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:19,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567839455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,489 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:19,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:19,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:19,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:19,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:19,492 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:19,492 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:19,492 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:19,492 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:19,493 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,493 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,494 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:19,494 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:19,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:19,494 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,494 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:19,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:19,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:19,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:19,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:19,495 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:19,495 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:19,495 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:19,496 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:19,496 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:19,496 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:19,496 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:19,496 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:19,496 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:19,496 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:19,496 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:19,496 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,496 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:19,496 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:19,497 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:19,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:19,497 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:19,498 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,498 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:19,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:19,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:19,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:19,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18675773[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:19,499 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:19,499 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,500 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,500 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:19,500 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,500 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:19,500 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:19,500 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:19,502 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,502 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:19,502 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:19,502 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:19,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:19,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:19,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:19,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:19,504 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:19,504 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:19,504 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:19,504 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:19,504 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:19,506 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:19,506 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,506 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:19,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:19,506 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:19,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,508 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:19,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,517 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:19,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:19,517 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:19,518 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:19,518 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:19,517 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:19,517 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:19,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:19,518 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:19,518 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,518 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,518 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,518 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,518 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:19,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:19,618 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:19,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:19,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:19,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:19,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:19,621 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:19,621 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:19,621 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:19,621 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:19,622 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,622 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,624 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,624 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:19,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:19,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:19,624 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,624 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:19,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:19,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:19,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:19,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:19,625 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:19,625 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:19,625 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:19,625 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:19,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:19,626 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:19,626 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:19,626 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:19,626 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:19,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:19,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:19,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:19,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:19,628 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,628 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:19,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:19,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:19,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:19,628 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,628 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:19,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:19,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b63b1da[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:19,629 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:19,629 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,631 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,631 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:19,631 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:19,631 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:19,631 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,631 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:19,640 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:19,640 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:19,640 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,640 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:19,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:19,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:19,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:19,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:19,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:19,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:19,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:19,643 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:19,643 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:19,643 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:19,643 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:19,643 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:19,646 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:19,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:19,646 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,646 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:19,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:19,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:19,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:19,646 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:19,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,648 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:19,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:19,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,651 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,651 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,651 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,651 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,651 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:19,651 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:19,651 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:19,651 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,651 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:19,651 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:19,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:19,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:19,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:19,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:19,651 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:19,651 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:19,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:19,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:19,667 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=734 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/1265458bb748416bb266af93e998f1f2 2023-04-27 03:56:19,677 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=734 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/3e85f1be105f456b8c71283e6a946871 2023-04-27 03:56:19,680 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/a91e43513d9741c191808d3201eddfbe as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/a91e43513d9741c191808d3201eddfbe 2023-04-27 03:56:19,683 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/a91e43513d9741c191808d3201eddfbe, entries=200, sequenceid=734, filesize=32.3 K 2023-04-27 03:56:19,684 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/1265458bb748416bb266af93e998f1f2 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/1265458bb748416bb266af93e998f1f2 2023-04-27 03:56:19,687 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/1265458bb748416bb266af93e998f1f2, entries=150, sequenceid=734, filesize=11.9 K 2023-04-27 03:56:19,687 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/3e85f1be105f456b8c71283e6a946871 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/3e85f1be105f456b8c71283e6a946871 2023-04-27 03:56:19,690 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/3e85f1be105f456b8c71283e6a946871, entries=150, sequenceid=734, filesize=11.9 K 2023-04-27 03:56:19,691 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 1c25ca7c0495563171ae7ccdf0350231 in 856ms, sequenceid=734, compaction requested=true 2023-04-27 03:56:19,691 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:19,691 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:19,691 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:19,691 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:56:19,691 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:56:19,691 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:19,692 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 79084 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:56:19,692 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35068 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:56:19,692 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/A is initiating minor compaction (all files) 2023-04-27 03:56:19,692 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/B is initiating minor compaction (all files) 2023-04-27 03:56:19,692 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/A in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:19,692 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/B in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:19,692 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/c47f0f651ec9463c922f9fc107748a13, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/7e149abbc3d442c893cedd5e7bb51d57, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/a91e43513d9741c191808d3201eddfbe] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=77.2 K 2023-04-27 03:56:19,692 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/c6f77fb963ca44e182608848d1180758, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/59f63a26c64b4bbfbdf339ae5f4620f8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/1265458bb748416bb266af93e998f1f2] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=34.2 K 2023-04-27 03:56:19,692 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting c47f0f651ec9463c922f9fc107748a13, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=684, earliestPutTs=1682567777335 2023-04-27 03:56:19,692 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting c6f77fb963ca44e182608848d1180758, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=684, earliestPutTs=1682567777335 2023-04-27 03:56:19,693 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 7e149abbc3d442c893cedd5e7bb51d57, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1682567777358 2023-04-27 03:56:19,693 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 59f63a26c64b4bbfbdf339ae5f4620f8, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1682567777358 2023-04-27 03:56:19,693 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting a91e43513d9741c191808d3201eddfbe, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=734, earliestPutTs=1682567778523 2023-04-27 03:56:19,693 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 1265458bb748416bb266af93e998f1f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=734, earliestPutTs=1682567778523 2023-04-27 03:56:19,705 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#B#compaction#608 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:19,716 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#A#compaction#607 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:19,731 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/a82ae29c035c458a8d8b462414921672 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/a82ae29c035c458a8d8b462414921672 2023-04-27 03:56:19,737 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/B of 1c25ca7c0495563171ae7ccdf0350231 into a82ae29c035c458a8d8b462414921672(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:19,737 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:19,737 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/B, priority=13, startTime=1682567779691; duration=0sec 2023-04-27 03:56:19,737 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:19,737 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:56:19,738 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35034 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:56:19,738 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/C is initiating minor compaction (all files) 2023-04-27 03:56:19,739 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/C in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:19,739 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/042d4cd9393d424fae7a24be13462db3, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/1aa21016a1b64319b4275be3de59f242, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/3e85f1be105f456b8c71283e6a946871] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=34.2 K 2023-04-27 03:56:19,740 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 042d4cd9393d424fae7a24be13462db3, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=684, earliestPutTs=1682567777335 2023-04-27 03:56:19,740 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 1aa21016a1b64319b4275be3de59f242, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1682567777358 2023-04-27 03:56:19,741 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 3e85f1be105f456b8c71283e6a946871, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=734, earliestPutTs=1682567778523 2023-04-27 03:56:19,743 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/8d2889461cf74684a7fc54f511c18800 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/8d2889461cf74684a7fc54f511c18800 2023-04-27 03:56:19,748 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/A of 1c25ca7c0495563171ae7ccdf0350231 into 8d2889461cf74684a7fc54f511c18800(size=26.5 K), total size for store is 26.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:19,748 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:19,748 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/A, priority=13, startTime=1682567779691; duration=0sec 2023-04-27 03:56:19,748 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:19,750 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#C#compaction#609 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:19,751 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:19,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:19,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:19,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:19,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:19,756 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:19,756 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:19,756 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:19,756 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:19,756 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,756 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,757 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,757 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:19,757 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/dac05457821b4707a9bc14c3ad05781d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/dac05457821b4707a9bc14c3ad05781d 2023-04-27 03:56:19,758 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:19,758 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:19,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:19,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:19,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:19,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:19,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:19,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:19,759 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:19,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:19,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:19,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:19,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:19,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:19,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:19,760 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:19,760 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:19,760 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-27 03:56:19,760 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:19,760 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:19,760 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:19,761 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:19,761 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:19,761 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:19,764 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/C of 1c25ca7c0495563171ae7ccdf0350231 into dac05457821b4707a9bc14c3ad05781d(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:19,764 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:19,764 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/C, priority=13, startTime=1682567779691; duration=0sec 2023-04-27 03:56:19,764 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:19,774 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042710237b1c662d423c95691d7496b51531 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042710237b1c662d423c95691d7496b51531 2023-04-27 03:56:19,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:19,958 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:19,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567779970, max timestamp=1682567779978Num uniques 859; 2023-04-27 03:56:19,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567779970, max timestamp=1682567779978Num uniques 859; 2023-04-27 03:56:19,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:56:19,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682567779970, max timestamp=1682567779978Num uniques 859; 2023-04-27 03:56:19,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:56:19,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-27 03:56:20,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682567780010, max timestamp=1682567780019Num uniques 1068; 2023-04-27 03:56:20,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682567780010, max timestamp=1682567780019Num uniques 1068; 2023-04-27 03:56:20,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-27 03:56:20,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:20,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682567780010, max timestamp=1682567780019Num uniques 1068; 2023-04-27 03:56:20,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567840019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-27 03:56:20,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-27 03:56:20,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:20,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567840043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:20,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 396 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567840044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:20,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567840123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:20,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567840147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:20,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 398 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567840147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,185 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=745, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/ee4b92ebd1234f17aab6243f8f9f97d3 2023-04-27 03:56:20,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:20,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567840327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:20,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567840351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:20,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567840351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,599 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=745 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/38c018d2a1fc4b68b7d276b432fabf25 2023-04-27 03:56:20,608 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=745 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/3719aa86c32440cd9b2dcd461bb03eaa 2023-04-27 03:56:20,611 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/ee4b92ebd1234f17aab6243f8f9f97d3 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/ee4b92ebd1234f17aab6243f8f9f97d3 2023-04-27 03:56:20,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:20,614 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/ee4b92ebd1234f17aab6243f8f9f97d3, entries=150, sequenceid=745, filesize=25.4 K 2023-04-27 03:56:20,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48474 deadline: 1682567840613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,616 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/38c018d2a1fc4b68b7d276b432fabf25 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/38c018d2a1fc4b68b7d276b432fabf25 2023-04-27 03:56:20,616 DEBUG [Thread-4053] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18266 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., hostname=jenkins-hbase4.apache.org,40885,1682567618305, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-27 03:56:20,619 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/38c018d2a1fc4b68b7d276b432fabf25, entries=150, sequenceid=745, filesize=11.9 K 2023-04-27 03:56:20,620 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/3719aa86c32440cd9b2dcd461bb03eaa as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/3719aa86c32440cd9b2dcd461bb03eaa 2023-04-27 03:56:20,623 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/3719aa86c32440cd9b2dcd461bb03eaa, entries=150, sequenceid=745, filesize=11.9 K 2023-04-27 03:56:20,624 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 1c25ca7c0495563171ae7ccdf0350231 in 864ms, sequenceid=745, compaction requested=false 2023-04-27 03:56:20,624 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:20,624 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:20,624 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:20,624 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:20,624 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,625 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:20,625 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:20,626 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:20,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:20,626 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:20,627 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:20,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:20,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:20,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:20,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:20,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c0d6fbd[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:20,628 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:20,628 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:20,630 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:20,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:20,630 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:20,630 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:20,631 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:20,631 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,631 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:20,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:20,631 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=147.60 KB heapSize=414.61 KB 2023-04-27 03:56:20,631 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:20,631 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:20,631 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:20,632 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:20,632 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:20,632 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:20,632 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:20,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:20,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:20,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:20,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:20,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:20,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:20,636 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:20,636 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:20,636 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:20,636 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:20,636 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:20,637 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,637 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:20,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,637 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:20,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:20,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:20,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:20,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:20,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,638 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:20,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:20,640 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:20,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:20,642 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:20,642 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:20,642 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:20,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:20,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:20,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:20,642 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,642 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:20,642 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:20,642 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:20,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:20,642 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:20,643 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,643 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:20,643 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:20,643 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:20,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:20,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:20,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567840652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:20,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567840653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:20,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567840655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,742 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:20,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:20,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:20,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:20,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:20,746 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:20,746 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:20,746 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:20,746 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:20,747 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,747 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:20,749 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:20,749 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:20,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:20,750 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,750 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:20,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:20,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:20,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:20,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:20,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:20,751 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:20,751 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:20,751 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:20,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:20,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567840755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:20,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:20,760 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:20,760 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:20,760 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:20,760 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:20,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:20,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:20,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:20,760 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:20,762 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,762 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:20,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:20,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:20,762 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:20,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:20,762 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:20,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:20,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39f475d6[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:20,764 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:20,764 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:20,766 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:20,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:20,766 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,766 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:20,766 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:20,766 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:20,766 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:20,767 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,767 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:20,767 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:20,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:20,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:20,767 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:20,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:20,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:20,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:20,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:20,770 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:20,770 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:20,770 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:20,770 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:20,770 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:20,772 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,772 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,773 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:20,773 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:20,774 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:20,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:20,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:20,776 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,777 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:20,777 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:20,777 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:20,777 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:20,777 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:20,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:20,777 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:20,777 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:20,777 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:20,777 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:20,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:20,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:20,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:20,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:20,777 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:20,777 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:20,878 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:20,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:20,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:20,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:20,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:20,881 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:20,881 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:20,881 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:20,881 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:20,881 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,882 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:20,886 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,886 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:20,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:20,886 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,886 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:20,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:20,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:20,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:20,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:20,887 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:20,887 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:20,887 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:20,888 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:20,888 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:20,888 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:20,888 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:20,888 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:20,888 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:20,888 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:20,888 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:20,888 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,888 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:20,888 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:20,891 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,891 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:20,892 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:20,892 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:20,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:20,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:20,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:20,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:20,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:20,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b0a6154[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:20,893 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:20,893 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:20,894 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:20,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:20,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:20,894 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:20,894 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:20,894 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:20,895 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,895 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:20,896 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,896 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:20,896 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:20,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:20,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:20,896 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:20,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:20,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:20,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:20,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:20,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:20,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:20,899 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:20,899 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:20,899 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:20,899 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:20,899 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:20,901 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,901 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,901 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:20,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:20,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:20,908 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:20,910 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,913 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:20,913 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:20,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:20,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:20,914 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,914 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:20,914 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:20,914 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:20,914 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:20,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:20,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:20,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:20,914 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:20,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:20,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:20,914 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:20,914 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:20,915 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:20,915 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:20,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:20,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567840957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,015 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:21,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:21,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:21,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:21,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:21,018 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:21,018 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:21,018 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:21,018 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:21,018 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,018 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:21,020 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:21,020 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:21,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:21,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:21,020 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,020 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:21,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:21,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:21,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:21,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:21,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:21,021 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:21,021 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:21,021 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:21,021 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:21,021 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:21,021 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:21,021 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:21,021 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:21,021 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:21,021 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:21,022 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:21,022 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,022 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:21,022 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:21,023 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,023 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:21,023 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,024 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:21,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:21,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:21,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:21,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d31ee8e[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:21,025 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:21,026 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,027 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,027 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:21,027 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:21,027 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:21,027 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,027 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:21,030 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:21,030 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:21,030 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,030 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:21,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:21,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:21,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:21,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:21,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:21,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:21,032 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:21,032 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:21,032 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:21,032 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:21,032 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:21,034 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,034 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:21,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,034 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:21,035 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:21,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:21,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:21,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:21,036 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:21,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,038 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:21,038 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,038 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:21,038 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:21,038 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:21,038 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,038 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,038 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:21,038 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:21,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:21,038 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:21,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:21,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:21,038 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:21,039 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:21,049 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427f8c8cc52dd0b410781114494724a0fcd to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427f8c8cc52dd0b410781114494724a0fcd 2023-04-27 03:56:21,139 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:21,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:21,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:21,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:21,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:21,142 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:21,142 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:21,142 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:21,142 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:21,142 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,143 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:21,144 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:21,144 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:21,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:21,144 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,144 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:21,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:21,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:21,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:21,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:21,145 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:21,145 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:21,145 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:21,145 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:21,145 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:21,146 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:21,146 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:21,146 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:21,146 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:21,146 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:21,146 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:21,146 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:21,146 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:21,146 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,146 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:21,146 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:21,154 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,154 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:21,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:21,154 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,154 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:21,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:21,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:21,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:21,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:21,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:21,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a060012[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:21,156 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:21,156 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,157 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,157 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:21,157 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:21,157 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:21,157 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,157 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:21,160 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,160 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:21,160 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:21,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:21,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:21,160 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:21,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:21,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567841160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:21,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567841162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:21,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:21,163 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:21,163 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:21,163 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:21,163 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:21,163 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:21,165 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,165 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:21,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:21,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,165 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:21,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:21,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:21,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:21,165 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:21,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:21,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:21,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,167 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:21,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:21,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,169 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,169 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:21,169 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:21,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:21,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:21,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:21,169 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:21,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:21,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:21,169 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:21,170 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:21,170 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:21,169 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:21,170 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:21,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:21,170 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,170 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,170 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:21,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567841262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,270 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:21,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:21,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:21,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:21,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:21,273 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:21,273 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:21,273 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:21,273 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:21,274 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,274 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:21,275 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,275 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:21,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:21,275 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,275 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:21,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:21,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:21,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:21,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:21,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:21,276 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:21,276 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:21,276 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:21,276 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:21,276 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:21,276 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:21,276 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:21,277 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:21,277 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:21,277 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:21,277 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:21,277 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,277 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:21,277 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:21,278 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:21,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:21,278 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:21,279 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,279 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:21,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:21,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:21,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:21,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:21,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10c68e41[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:21,280 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:21,280 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,285 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,286 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:21,286 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:21,286 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:21,286 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,286 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:21,287 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,287 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:21,287 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:21,287 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:21,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:21,290 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:21,290 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:21,290 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:21,290 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:21,290 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:21,291 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,291 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:21,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,291 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:21,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:21,292 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:21,293 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,295 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,295 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:21,295 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:21,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:21,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:21,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:21,295 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:21,295 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:21,295 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:21,296 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,296 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,296 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,296 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,296 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:21,296 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:21,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:21,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:21,396 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:21,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:21,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:21,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:21,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:21,399 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:21,399 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:21,400 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:21,400 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:21,400 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,400 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:21,402 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:21,402 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:21,402 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,402 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:21,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:21,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:21,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:21,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:21,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:21,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:21,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:21,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:21,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:21,404 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:21,404 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:21,404 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:21,404 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:21,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:21,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:21,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,404 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:21,405 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:21,406 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:21,406 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:21,407 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,407 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:21,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:21,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@588e4dcc[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:21,408 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:21,408 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,409 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,409 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,409 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:21,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,410 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:21,410 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:21,410 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:21,411 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:21,411 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:21,411 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,411 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:21,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:21,414 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:21,414 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:21,414 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:21,414 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:21,414 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:21,419 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:21,419 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,419 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:21,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:21,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:21,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:21,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:21,420 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:21,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:21,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,421 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:21,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:21,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,423 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:21,424 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:21,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:21,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:21,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:21,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:21,424 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:21,424 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,424 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:21,424 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:21,424 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,424 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,424 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:21,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:21,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:21,424 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,428 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:21,428 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:21,454 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=800, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/f12f4f70032d498faccdd53ce5b28618 2023-04-27 03:56:21,463 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=800 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/47788a00420949bc87ce07ad86cc8908 2023-04-27 03:56:21,472 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=800 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/a6825d05a55d4892ba69427739186db0 2023-04-27 03:56:21,475 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/f12f4f70032d498faccdd53ce5b28618 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/f12f4f70032d498faccdd53ce5b28618 2023-04-27 03:56:21,479 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/f12f4f70032d498faccdd53ce5b28618, entries=150, sequenceid=800, filesize=25.4 K 2023-04-27 03:56:21,480 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/47788a00420949bc87ce07ad86cc8908 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/47788a00420949bc87ce07ad86cc8908 2023-04-27 03:56:21,483 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/47788a00420949bc87ce07ad86cc8908, entries=150, sequenceid=800, filesize=11.9 K 2023-04-27 03:56:21,483 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/a6825d05a55d4892ba69427739186db0 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a6825d05a55d4892ba69427739186db0 2023-04-27 03:56:21,486 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a6825d05a55d4892ba69427739186db0, entries=150, sequenceid=800, filesize=11.9 K 2023-04-27 03:56:21,487 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~414.56 KB/424512, currentSize=33.54 KB/34350 for 1c25ca7c0495563171ae7ccdf0350231 in 856ms, sequenceid=800, compaction requested=true 2023-04-27 03:56:21,487 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:21,487 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:21,487 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:56:21,487 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:56:21,487 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:21,487 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:21,488 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 79194 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:56:21,488 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/A is initiating minor compaction (all files) 2023-04-27 03:56:21,488 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/A in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:21,488 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:56:21,488 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/8d2889461cf74684a7fc54f511c18800, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/ee4b92ebd1234f17aab6243f8f9f97d3, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/f12f4f70032d498faccdd53ce5b28618] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=77.3 K 2023-04-27 03:56:21,488 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/B is initiating minor compaction (all files) 2023-04-27 03:56:21,488 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/B in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:21,488 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/a82ae29c035c458a8d8b462414921672, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/38c018d2a1fc4b68b7d276b432fabf25, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/47788a00420949bc87ce07ad86cc8908] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=36.7 K 2023-04-27 03:56:21,489 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 8d2889461cf74684a7fc54f511c18800, keycount=150, bloomtype=ROW, size=26.5 K, encoding=NONE, compression=NONE, seqNum=734, earliestPutTs=1682567778523 2023-04-27 03:56:21,489 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting a82ae29c035c458a8d8b462414921672, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=734, earliestPutTs=1682567778523 2023-04-27 03:56:21,489 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting ee4b92ebd1234f17aab6243f8f9f97d3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=745, earliestPutTs=1682567778837 2023-04-27 03:56:21,489 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 38c018d2a1fc4b68b7d276b432fabf25, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=745, earliestPutTs=1682567778837 2023-04-27 03:56:21,489 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting f12f4f70032d498faccdd53ce5b28618, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=800, earliestPutTs=1682567780039 2023-04-27 03:56:21,489 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 47788a00420949bc87ce07ad86cc8908, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=800, earliestPutTs=1682567780039 2023-04-27 03:56:21,497 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#B#compaction#617 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:21,498 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#A#compaction#616 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:21,528 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:21,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:21,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:21,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:21,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:21,536 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:21,536 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:21,536 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:21,536 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:21,537 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,537 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:21,538 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,538 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:21,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:21,538 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:21,538 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:21,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:21,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:21,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:21,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:21,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:21,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:21,545 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:21,545 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:21,545 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:21,545 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:21,545 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:21,545 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:21,545 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:21,555 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-27 03:56:21,556 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:21,556 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:21,556 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:21,556 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:21,556 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:21,556 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:21,572 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427308f356e5d094e7f93fd66ad66561f74 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427308f356e5d094e7f93fd66ad66561f74 2023-04-27 03:56:21,577 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=808, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/45047dce38654811a1c834e114f79389 2023-04-27 03:56:21,598 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=808 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/4dba49bc6b01417a87c9aa8b9c2d46c3 2023-04-27 03:56:21,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:21,772 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:21,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567781767, max timestamp=1682567781820Num uniques 859; 2023-04-27 03:56:21,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:56:21,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567781767, max timestamp=1682567781820Num uniques 859; 2023-04-27 03:56:21,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682567781767, max timestamp=1682567781820Num uniques 859; 2023-04-27 03:56:21,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:56:21,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-27 03:56:21,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682567781868, max timestamp=1682567781878Num uniques 1018; 2023-04-27 03:56:21,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-27 03:56:21,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682567781868, max timestamp=1682567781878Num uniques 1018; 2023-04-27 03:56:21,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682567781868, max timestamp=1682567781878Num uniques 1018; 2023-04-27 03:56:21,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-27 03:56:21,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-27 03:56:21,905 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/0becb8ea67bb489fb4111fc6ee262502 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/0becb8ea67bb489fb4111fc6ee262502 2023-04-27 03:56:21,910 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/B of 1c25ca7c0495563171ae7ccdf0350231 into 0becb8ea67bb489fb4111fc6ee262502(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:21,910 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:21,910 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/B, priority=13, startTime=1682567781487; duration=0sec 2023-04-27 03:56:21,910 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:21,910 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:56:21,911 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:56:21,911 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/C is initiating minor compaction (all files) 2023-04-27 03:56:21,911 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/C in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:21,911 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/dac05457821b4707a9bc14c3ad05781d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/3719aa86c32440cd9b2dcd461bb03eaa, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a6825d05a55d4892ba69427739186db0] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=36.7 K 2023-04-27 03:56:21,911 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting dac05457821b4707a9bc14c3ad05781d, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=734, earliestPutTs=1682567778523 2023-04-27 03:56:21,912 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 3719aa86c32440cd9b2dcd461bb03eaa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=745, earliestPutTs=1682567778837 2023-04-27 03:56:21,912 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting a6825d05a55d4892ba69427739186db0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=800, earliestPutTs=1682567780039 2023-04-27 03:56:21,917 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#C#compaction#621 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:21,921 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/611a9e30647e439b903535290cb0673f as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/611a9e30647e439b903535290cb0673f 2023-04-27 03:56:21,925 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/A of 1c25ca7c0495563171ae7ccdf0350231 into 611a9e30647e439b903535290cb0673f(size=26.6 K), total size for store is 26.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:21,925 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:21,925 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/A, priority=13, startTime=1682567781487; duration=0sec 2023-04-27 03:56:21,925 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:21,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682567781907, max timestamp=1682567781935Num uniques 1027; 2023-04-27 03:56:21,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-27 03:56:21,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682567781907, max timestamp=1682567781935Num uniques 1027; 2023-04-27 03:56:21,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682567781907, max timestamp=1682567781935Num uniques 1027; 2023-04-27 03:56:21,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-27 03:56:21,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-27 03:56:21,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682567781972, max timestamp=1682567781990Num uniques 1036; 2023-04-27 03:56:21,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-27 03:56:21,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682567781972, max timestamp=1682567781990Num uniques 1036; 2023-04-27 03:56:21,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682567781972, max timestamp=1682567781990Num uniques 1036; 2023-04-27 03:56:21,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-27 03:56:21,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-27 03:56:22,014 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=808 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/17b98913bab1492f8c44f73f190e382d 2023-04-27 03:56:22,018 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/45047dce38654811a1c834e114f79389 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/45047dce38654811a1c834e114f79389 2023-04-27 03:56:22,021 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/45047dce38654811a1c834e114f79389, entries=100, sequenceid=808, filesize=18.6 K 2023-04-27 03:56:22,022 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/4dba49bc6b01417a87c9aa8b9c2d46c3 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/4dba49bc6b01417a87c9aa8b9c2d46c3 2023-04-27 03:56:22,025 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/4dba49bc6b01417a87c9aa8b9c2d46c3, entries=100, sequenceid=808, filesize=9.5 K 2023-04-27 03:56:22,026 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/17b98913bab1492f8c44f73f190e382d as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/17b98913bab1492f8c44f73f190e382d 2023-04-27 03:56:22,030 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/17b98913bab1492f8c44f73f190e382d, entries=100, sequenceid=808, filesize=9.5 K 2023-04-27 03:56:22,030 INFO [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=114.05 KB/116790 for 1c25ca7c0495563171ae7ccdf0350231 in 475ms, sequenceid=808, compaction requested=false 2023-04-27 03:56:22,031 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:22,031 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:22,031 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:22,031 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:22,031 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,031 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:22,031 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:22,032 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:22,032 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:22,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:22,033 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:22,033 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:22,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:22,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:22,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:22,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:22,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:22,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a899dcd[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:22,034 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:22,034 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:22,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:22,035 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=120.76 KB heapSize=337.92 KB 2023-04-27 03:56:22,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:22,036 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:22,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:22,036 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:22,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:22,036 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-27 03:56:22,039 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:22,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:22,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:22,039 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:22,039 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:22,039 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,039 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:22,039 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:22,040 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,040 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:22,041 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:22,041 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:22,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:22,044 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:22,044 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:22,044 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:22,044 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:22,044 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:22,046 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,046 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,046 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:22,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:22,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,047 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:22,048 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,049 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427d33efab45de248c9819a89c2a43a3722 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427d33efab45de248c9819a89c2a43a3722 2023-04-27 03:56:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,052 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,052 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:22,052 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:22,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:22,052 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:22,052 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:22,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:22,052 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:22,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:22,053 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:22,053 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:22,053 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,053 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:22,053 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:22,054 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=895, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/d7cee918e0de4fea82cdae8b6a6b1643 2023-04-27 03:56:22,052 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:22,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:22,065 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=895 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/bbbf65d961a04fdbaf0684a2366d99ec 2023-04-27 03:56:22,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:22,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567842073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,153 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:22,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:22,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:22,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:22,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:22,161 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:22,161 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:22,161 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:22,161 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:22,162 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,162 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:22,163 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:22,163 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:22,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:22,164 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,164 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:22,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:22,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:22,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:22,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:22,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:22,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:22,165 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:22,165 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:22,165 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:22,165 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:22,165 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:22,165 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:22,165 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:22,165 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:22,165 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:22,166 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:22,166 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:22,166 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,166 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:22,166 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:22,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:22,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48458 deadline: 1682567842163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,167 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,167 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:22,167 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:22,167 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:22,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:22,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40885] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48502 deadline: 1682567842165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:22,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:22,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2035081e[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:22,169 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:22,169 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:22,170 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:22,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:22,170 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:22,170 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:22,170 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,170 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:22,170 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:22,179 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:22,179 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,179 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:22,179 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:22,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:22,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40885] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567842177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:22,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:22,181 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:22,181 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:22,181 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:22,181 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:22,181 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:22,183 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,183 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:22,183 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:22,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:22,183 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:22,185 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,187 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:22,187 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:22,187 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:22,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:22,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:22,187 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:22,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:22,187 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:22,187 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:22,187 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,187 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:22,187 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:22,187 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,187 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:22,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:22,188 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:22,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:22,288 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:22,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:22,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-27 03:56:22,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-27 03:56:22,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-27 03:56:22,291 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-27 03:56:22,291 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:22,291 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-27 03:56:22,291 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-27 03:56:22,291 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,291 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:22,292 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:22,292 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:22,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:22,293 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,293 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-27 03:56:22,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:22,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-27 03:56:22,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:22,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-27 03:56:22,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-27 03:56:22,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-27 03:56:22,294 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-27 03:56:22,294 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-27 03:56:22,294 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-27 03:56:22,294 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-27 03:56:22,294 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-27 03:56:22,294 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:22,294 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. started... 2023-04-27 03:56:22,294 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. as already flushing 2023-04-27 03:56:22,294 DEBUG [rs(jenkins-hbase4.apache.org,40885,1682567618305)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:22,294 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-27 03:56:22,294 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-27 03:56:22,294 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,295 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-27 03:56:22,295 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-27 03:56:22,297 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:22,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:22,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:22,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:22,297 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:22,298 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:22,298 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-27 03:56:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:22,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40885,1682567618305' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-27 03:56:22,298 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-27 03:56:22,299 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:22,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a52edf[Count = 0] remaining members to acquire global barrier 2023-04-27 03:56:22,300 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:22,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:22,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:22,300 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-27 03:56:22,300 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-27 03:56:22,300 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40885,1682567618305' in zk 2023-04-27 03:56:22,300 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,300 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-27 03:56:22,301 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,301 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-27 03:56:22,301 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:22,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:22,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:22,301 DEBUG [member: 'jenkins-hbase4.apache.org,40885,1682567618305' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-27 03:56:22,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:22,302 DEBUG [Thread-4060] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07d997c6 to 127.0.0.1:55020 2023-04-27 03:56:22,302 DEBUG [Thread-4060] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:56:22,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:22,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:22,305 DEBUG [Thread-4058] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f4eab6d to 127.0.0.1:55020 2023-04-27 03:56:22,306 DEBUG [Thread-4058] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:56:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,306 DEBUG [Thread-4064] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fae3ed2 to 127.0.0.1:55020 2023-04-27 03:56:22,306 DEBUG [Thread-4064] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:56:22,307 DEBUG [Thread-4062] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7993d99e to 127.0.0.1:55020 2023-04-27 03:56:22,307 DEBUG [Thread-4062] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:56:22,308 DEBUG [Thread-4056] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66c25427 to 127.0.0.1:55020 2023-04-27 03:56:22,308 DEBUG [Thread-4056] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:56:22,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40885,1682567618305': 2023-04-27 03:56:22,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40885,1682567618305' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-27 03:56:22,313 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-27 03:56:22,313 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-27 03:56:22,313 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-27 03:56:22,313 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-27 03:56:22,313 INFO [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-27 03:56:22,314 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,314 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-27 03:56:22,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-27 03:56:22,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,314 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-27 03:56:22,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:22,315 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-27 03:56:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:22,316 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-27 03:56:22,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-27 03:56:22,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,318 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,318 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:22,318 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-27 03:56:22,318 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-27 03:56:22,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-27 03:56:22,318 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-27 03:56:22,318 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:22,318 DEBUG [(jenkins-hbase4.apache.org,39131,1682567617081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-27 03:56:22,318 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-27 03:56:22,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-27 03:56:22,318 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-27 03:56:22,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-27 03:56:22,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-27 03:56:22,318 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-27 03:56:22,318 DEBUG [Thread-4055] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-27 03:56:22,318 DEBUG [Thread-4055] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-27 03:56:22,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-27 03:56:22,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-27 03:56:22,325 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/36af05db406d492abe6f4d3c7281508c as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/36af05db406d492abe6f4d3c7281508c 2023-04-27 03:56:22,327 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/C of 1c25ca7c0495563171ae7ccdf0350231 into 36af05db406d492abe6f4d3c7281508c(size=13.0 K), total size for store is 22.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:22,327 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:22,327 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/C, priority=13, startTime=1682567781487; duration=0sec 2023-04-27 03:56:22,328 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:22,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-27 03:56:22,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48534 deadline: 1682567842381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:22,418 DEBUG [Thread-4055] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-27 03:56:22,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-27 03:56:22,474 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=895 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/e404a79f5d934dea96374c661f25c760 2023-04-27 03:56:22,479 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/d7cee918e0de4fea82cdae8b6a6b1643 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/d7cee918e0de4fea82cdae8b6a6b1643 2023-04-27 03:56:22,482 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/d7cee918e0de4fea82cdae8b6a6b1643, entries=200, sequenceid=895, filesize=32.3 K 2023-04-27 03:56:22,482 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/bbbf65d961a04fdbaf0684a2366d99ec as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/bbbf65d961a04fdbaf0684a2366d99ec 2023-04-27 03:56:22,485 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/bbbf65d961a04fdbaf0684a2366d99ec, entries=150, sequenceid=895, filesize=11.9 K 2023-04-27 03:56:22,485 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/e404a79f5d934dea96374c661f25c760 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/e404a79f5d934dea96374c661f25c760 2023-04-27 03:56:22,488 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/e404a79f5d934dea96374c661f25c760, entries=150, sequenceid=895, filesize=11.9 K 2023-04-27 03:56:22,489 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~337.88 KB/345984, currentSize=60.38 KB/61830 for 1c25ca7c0495563171ae7ccdf0350231 in 454ms, sequenceid=895, compaction requested=true 2023-04-27 03:56:22,489 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:22,489 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:22,489 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:22,489 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:56:22,489 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:56:22,489 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:22,490 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 79288 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:56:22,490 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/A is initiating minor compaction (all files) 2023-04-27 03:56:22,490 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35272 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:56:22,490 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/A in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:22,490 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/B is initiating minor compaction (all files) 2023-04-27 03:56:22,490 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/611a9e30647e439b903535290cb0673f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/45047dce38654811a1c834e114f79389, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/d7cee918e0de4fea82cdae8b6a6b1643] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=77.4 K 2023-04-27 03:56:22,490 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/B in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:22,490 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/0becb8ea67bb489fb4111fc6ee262502, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/4dba49bc6b01417a87c9aa8b9c2d46c3, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/bbbf65d961a04fdbaf0684a2366d99ec] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=34.4 K 2023-04-27 03:56:22,490 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 611a9e30647e439b903535290cb0673f, keycount=150, bloomtype=ROW, size=26.6 K, encoding=NONE, compression=NONE, seqNum=800, earliestPutTs=1682567780039 2023-04-27 03:56:22,490 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 0becb8ea67bb489fb4111fc6ee262502, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=800, earliestPutTs=1682567780039 2023-04-27 03:56:22,491 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting 45047dce38654811a1c834e114f79389, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=808, earliestPutTs=1682567780648 2023-04-27 03:56:22,491 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 4dba49bc6b01417a87c9aa8b9c2d46c3, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=808, earliestPutTs=1682567780648 2023-04-27 03:56:22,491 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] compactions.Compactor(207): Compacting d7cee918e0de4fea82cdae8b6a6b1643, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=895, earliestPutTs=1682567782021 2023-04-27 03:56:22,491 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting bbbf65d961a04fdbaf0684a2366d99ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=895, earliestPutTs=1682567782021 2023-04-27 03:56:22,496 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#B#compaction#625 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:22,497 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#A#compaction#626 average throughput is 9.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:22,502 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/05983e22d37f41889ccf7e3946e9c1ee as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/05983e22d37f41889ccf7e3946e9c1ee 2023-04-27 03:56:22,505 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/B of 1c25ca7c0495563171ae7ccdf0350231 into 05983e22d37f41889ccf7e3946e9c1ee(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:22,505 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:22,505 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/B, priority=13, startTime=1682567782489; duration=0sec 2023-04-27 03:56:22,505 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-27 03:56:22,505 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-27 03:56:22,506 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-27 03:56:22,506 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1912): 1c25ca7c0495563171ae7ccdf0350231/C is initiating minor compaction (all files) 2023-04-27 03:56:22,506 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1c25ca7c0495563171ae7ccdf0350231/C in TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:22,506 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/36af05db406d492abe6f4d3c7281508c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/17b98913bab1492f8c44f73f190e382d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/e404a79f5d934dea96374c661f25c760] into tmpdir=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp, totalSize=34.4 K 2023-04-27 03:56:22,508 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 36af05db406d492abe6f4d3c7281508c, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=800, earliestPutTs=1682567780039 2023-04-27 03:56:22,508 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting 17b98913bab1492f8c44f73f190e382d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=808, earliestPutTs=1682567780648 2023-04-27 03:56:22,508 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] compactions.Compactor(207): Compacting e404a79f5d934dea96374c661f25c760, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=895, earliestPutTs=1682567782021 2023-04-27 03:56:22,513 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] throttle.PressureAwareThroughputController(145): 1c25ca7c0495563171ae7ccdf0350231#C#compaction#627 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-27 03:56:22,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40885] regionserver.HRegion(9158): Flush requested on 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:22,685 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-27 03:56:22,686 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:22,686 DEBUG [Thread-4049] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d759aab to 127.0.0.1:55020 2023-04-27 03:56:22,686 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:22,686 DEBUG [Thread-4049] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:56:22,686 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:22,686 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:22,686 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:22,686 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:22,696 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427c18d02f0503a49e6ac3c4ca39010afd5 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427c18d02f0503a49e6ac3c4ca39010afd5 2023-04-27 03:56:22,699 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=910, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/859f74522f9a4ac0b409c0833a77b861 2023-04-27 03:56:22,706 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=910 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/6fa5aa8ff3fe4ca2a625b958319df65b 2023-04-27 03:56:22,713 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=910 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/a107ce5e72264d288567b7c21ecb1146 2023-04-27 03:56:22,715 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/859f74522f9a4ac0b409c0833a77b861 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/859f74522f9a4ac0b409c0833a77b861 2023-04-27 03:56:22,718 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/859f74522f9a4ac0b409c0833a77b861, entries=150, sequenceid=910, filesize=25.4 K 2023-04-27 03:56:22,719 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/6fa5aa8ff3fe4ca2a625b958319df65b as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/6fa5aa8ff3fe4ca2a625b958319df65b 2023-04-27 03:56:22,721 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/6fa5aa8ff3fe4ca2a625b958319df65b, entries=150, sequenceid=910, filesize=11.9 K 2023-04-27 03:56:22,721 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/a107ce5e72264d288567b7c21ecb1146 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a107ce5e72264d288567b7c21ecb1146 2023-04-27 03:56:22,724 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a107ce5e72264d288567b7c21ecb1146, entries=150, sequenceid=910, filesize=11.9 K 2023-04-27 03:56:22,724 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 1c25ca7c0495563171ae7ccdf0350231 in 39ms, sequenceid=910, compaction requested=false 2023-04-27 03:56:22,724 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:22,919 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/0fa41d44e3514fd78b7de0dac19da3ca as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/0fa41d44e3514fd78b7de0dac19da3ca 2023-04-27 03:56:22,922 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/C of 1c25ca7c0495563171ae7ccdf0350231 into 0fa41d44e3514fd78b7de0dac19da3ca(size=13.1 K), total size for store is 25.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:22,922 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:22,922 INFO [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/C, priority=13, startTime=1682567782489; duration=0sec 2023-04-27 03:56:22,922 DEBUG [RS:0;jenkins-hbase4:40885-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:23,312 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/82aa26af80964c53b27c531e7110e0f5 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/82aa26af80964c53b27c531e7110e0f5 2023-04-27 03:56:23,316 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1c25ca7c0495563171ae7ccdf0350231/A of 1c25ca7c0495563171ae7ccdf0350231 into 82aa26af80964c53b27c531e7110e0f5(size=26.7 K), total size for store is 52.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-27 03:56:23,316 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:23,316 INFO [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231., storeName=1c25ca7c0495563171ae7ccdf0350231/A, priority=13, startTime=1682567782489; duration=0sec 2023-04-27 03:56:23,316 DEBUG [RS:0;jenkins-hbase4:40885-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-27 03:56:24,172 DEBUG [Thread-4051] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x39241fe3 to 127.0.0.1:55020 2023-04-27 03:56:24,172 DEBUG [Thread-4051] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:56:24,178 DEBUG [Thread-4045] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1644730f to 127.0.0.1:55020 2023-04-27 03:56:24,178 DEBUG [Thread-4045] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:56:28,037 DEBUG [Thread-4047] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6af74c0e to 127.0.0.1:55020 2023-04-27 03:56:28,037 DEBUG [Thread-4047] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:56:30,655 DEBUG [Thread-4053] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1fbb5b46 to 127.0.0.1:55020 2023-04-27 03:56:30,655 DEBUG [Thread-4053] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:56:30,656 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-27 03:56:30,656 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 187 2023-04-27 03:56:30,656 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 55 2023-04-27 03:56:30,656 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 253 2023-04-27 03:56:30,656 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 289 2023-04-27 03:56:30,656 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(394): wrote 6 2023-04-27 03:56:30,656 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-27 03:56:30,656 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(398): read 4513 2023-04-27 03:56:30,656 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(398): read 4531 2023-04-27 03:56:30,656 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(398): read 4649 2023-04-27 03:56:30,656 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(398): read 4525 2023-04-27 03:56:30,656 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(398): read 4518 2023-04-27 03:56:30,656 INFO [Listener at localhost/33933] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-27 03:56:30,656 INFO [Listener at localhost/33933] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-27 03:56:30,656 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x45878790 to 127.0.0.1:55020 2023-04-27 03:56:30,656 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:56:30,657 INFO [Listener at localhost/33933] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-27 03:56:30,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-27 03:56:30,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-27 03:56:30,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-27 03:56:30,660 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567790660"}]},"ts":"1682567790660"} 2023-04-27 03:56:30,661 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-27 03:56:30,662 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-27 03:56:30,663 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1c25ca7c0495563171ae7ccdf0350231, UNASSIGN}] 2023-04-27 03:56:30,664 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1c25ca7c0495563171ae7ccdf0350231, UNASSIGN 2023-04-27 03:56:30,664 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=1c25ca7c0495563171ae7ccdf0350231, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:30,665 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567790664"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682567790664"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682567790664"}]},"ts":"1682567790664"} 2023-04-27 03:56:30,665 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305}] 2023-04-27 03:56:30,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-27 03:56:30,817 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:30,817 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1c25ca7c0495563171ae7ccdf0350231, disabling compactions & flushes 2023-04-27 03:56:30,817 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:30,817 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:30,817 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. after waiting 0 ms 2023-04-27 03:56:30,817 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:30,817 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1c25ca7c0495563171ae7ccdf0350231 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-27 03:56:30,817 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=A 2023-04-27 03:56:30,817 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:30,817 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=B 2023-04-27 03:56:30,817 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:30,818 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1c25ca7c0495563171ae7ccdf0350231, store=C 2023-04-27 03:56:30,818 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-27 03:56:30,832 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230427eae0c32fc08b4f66b85c3b022e8fbe62 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427eae0c32fc08b4f66b85c3b022e8fbe62 2023-04-27 03:56:30,836 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=919, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/b04e2e99fab742f885a72a1c23fda382 2023-04-27 03:56:30,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-27 03:56:31,243 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=919 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/d79c39e970be44c4b0b280f1ca32954b 2023-04-27 03:56:31,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-27 03:56:31,651 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=919 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/ccd307361cf34cf58a3cda06abe22538 2023-04-27 03:56:31,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/A/b04e2e99fab742f885a72a1c23fda382 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/b04e2e99fab742f885a72a1c23fda382 2023-04-27 03:56:31,658 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/b04e2e99fab742f885a72a1c23fda382, entries=150, sequenceid=919, filesize=25.4 K 2023-04-27 03:56:31,658 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/B/d79c39e970be44c4b0b280f1ca32954b as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/d79c39e970be44c4b0b280f1ca32954b 2023-04-27 03:56:31,661 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/d79c39e970be44c4b0b280f1ca32954b, entries=150, sequenceid=919, filesize=11.9 K 2023-04-27 03:56:31,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/.tmp/C/ccd307361cf34cf58a3cda06abe22538 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/ccd307361cf34cf58a3cda06abe22538 2023-04-27 03:56:31,664 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/ccd307361cf34cf58a3cda06abe22538, entries=150, sequenceid=919, filesize=11.9 K 2023-04-27 03:56:31,665 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 1c25ca7c0495563171ae7ccdf0350231 in 848ms, sequenceid=919, compaction requested=true 2023-04-27 03:56:31,686 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/acfbeada10d34d91959f1b857d5a5f33, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/969653295d8e41039b0d3dc9085acd41, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/6f12e5f9e8f24cc994b2cc8629ad9403, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/fca51d0e7ade484b9d0cc1776d31dde9, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/7606d53b0e1a45c4a0dd8512512bb7cf, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/098b019827f54f72bcf8954f6d5ed54c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/3bd85fdf4a6e403688579f1915a6fa86, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/26341f746d034850862c6645b571232b, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/26ebed07f6934b2899d1e10234ced1ed, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/6b58444e2dbb4f56b8c8a5a1819272ba, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/9fc988a166f544ae9c2dcf6d5aa16381, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/dd922210d84c45b3a976705dc732d605, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/8dbcd37ab3e344d690faa1de723fad83, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/08ae9c0c6a484adebb1aebd4bf38d749, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/0d67b9ece64e42b89ffb2432501387c1, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/06bf6d3c060a406cbb8d9ab36841ed9c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/be9ea24a70f24feba5e4af74bae66136, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/fb2aaba0a5f34d469df22d4ff9ceca87, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/46bb7ae5d5e848c6bd202d79246b1575, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/29dca4d280b04757be7e16e7f1cedab9, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/23fc139379e54279a94ce7664e467a1e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/0699339f9efa4259bcd99cd6e856dd2f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/5273011fed834436ab8c27207725b7cb, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/e882a5288dd6476686eada4ca2f60ab4, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/1ad567a9e4b849c5b939ae8f061bf871, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/5034e204cc9d444ab0741ca8113d060e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/b74c4ee927fd47cdbe9df5b3fb42f695, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/aebc5e7cc2e64842a582a5c07ec7d2e5, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/c47f0f651ec9463c922f9fc107748a13, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/7e149abbc3d442c893cedd5e7bb51d57, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/a91e43513d9741c191808d3201eddfbe, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/8d2889461cf74684a7fc54f511c18800, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/ee4b92ebd1234f17aab6243f8f9f97d3, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/611a9e30647e439b903535290cb0673f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/f12f4f70032d498faccdd53ce5b28618, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/45047dce38654811a1c834e114f79389, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/d7cee918e0de4fea82cdae8b6a6b1643] to archive 2023-04-27 03:56:31,687 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 03:56:31,689 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/acfbeada10d34d91959f1b857d5a5f33 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/acfbeada10d34d91959f1b857d5a5f33 2023-04-27 03:56:31,690 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/969653295d8e41039b0d3dc9085acd41 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/969653295d8e41039b0d3dc9085acd41 2023-04-27 03:56:31,690 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/6f12e5f9e8f24cc994b2cc8629ad9403 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/6f12e5f9e8f24cc994b2cc8629ad9403 2023-04-27 03:56:31,691 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/fca51d0e7ade484b9d0cc1776d31dde9 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/fca51d0e7ade484b9d0cc1776d31dde9 2023-04-27 03:56:31,692 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/7606d53b0e1a45c4a0dd8512512bb7cf to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/7606d53b0e1a45c4a0dd8512512bb7cf 2023-04-27 03:56:31,693 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/098b019827f54f72bcf8954f6d5ed54c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/098b019827f54f72bcf8954f6d5ed54c 2023-04-27 03:56:31,693 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/3bd85fdf4a6e403688579f1915a6fa86 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/3bd85fdf4a6e403688579f1915a6fa86 2023-04-27 03:56:31,694 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/26341f746d034850862c6645b571232b to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/26341f746d034850862c6645b571232b 2023-04-27 03:56:31,695 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/26ebed07f6934b2899d1e10234ced1ed to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/26ebed07f6934b2899d1e10234ced1ed 2023-04-27 03:56:31,696 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/6b58444e2dbb4f56b8c8a5a1819272ba to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/6b58444e2dbb4f56b8c8a5a1819272ba 2023-04-27 03:56:31,697 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/9fc988a166f544ae9c2dcf6d5aa16381 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/9fc988a166f544ae9c2dcf6d5aa16381 2023-04-27 03:56:31,698 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/dd922210d84c45b3a976705dc732d605 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/dd922210d84c45b3a976705dc732d605 2023-04-27 03:56:31,698 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/8dbcd37ab3e344d690faa1de723fad83 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/8dbcd37ab3e344d690faa1de723fad83 2023-04-27 03:56:31,699 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/08ae9c0c6a484adebb1aebd4bf38d749 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/08ae9c0c6a484adebb1aebd4bf38d749 2023-04-27 03:56:31,700 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/0d67b9ece64e42b89ffb2432501387c1 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/0d67b9ece64e42b89ffb2432501387c1 2023-04-27 03:56:31,701 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/06bf6d3c060a406cbb8d9ab36841ed9c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/06bf6d3c060a406cbb8d9ab36841ed9c 2023-04-27 03:56:31,701 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/be9ea24a70f24feba5e4af74bae66136 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/be9ea24a70f24feba5e4af74bae66136 2023-04-27 03:56:31,702 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/fb2aaba0a5f34d469df22d4ff9ceca87 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/fb2aaba0a5f34d469df22d4ff9ceca87 2023-04-27 03:56:31,703 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/46bb7ae5d5e848c6bd202d79246b1575 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/46bb7ae5d5e848c6bd202d79246b1575 2023-04-27 03:56:31,704 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/29dca4d280b04757be7e16e7f1cedab9 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/29dca4d280b04757be7e16e7f1cedab9 2023-04-27 03:56:31,705 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/23fc139379e54279a94ce7664e467a1e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/23fc139379e54279a94ce7664e467a1e 2023-04-27 03:56:31,705 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/0699339f9efa4259bcd99cd6e856dd2f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/0699339f9efa4259bcd99cd6e856dd2f 2023-04-27 03:56:31,706 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/5273011fed834436ab8c27207725b7cb to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/5273011fed834436ab8c27207725b7cb 2023-04-27 03:56:31,707 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/e882a5288dd6476686eada4ca2f60ab4 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/e882a5288dd6476686eada4ca2f60ab4 2023-04-27 03:56:31,708 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/1ad567a9e4b849c5b939ae8f061bf871 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/1ad567a9e4b849c5b939ae8f061bf871 2023-04-27 03:56:31,708 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/5034e204cc9d444ab0741ca8113d060e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/5034e204cc9d444ab0741ca8113d060e 2023-04-27 03:56:31,709 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/b74c4ee927fd47cdbe9df5b3fb42f695 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/b74c4ee927fd47cdbe9df5b3fb42f695 2023-04-27 03:56:31,710 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/aebc5e7cc2e64842a582a5c07ec7d2e5 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/aebc5e7cc2e64842a582a5c07ec7d2e5 2023-04-27 03:56:31,711 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/c47f0f651ec9463c922f9fc107748a13 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/c47f0f651ec9463c922f9fc107748a13 2023-04-27 03:56:31,712 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/7e149abbc3d442c893cedd5e7bb51d57 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/7e149abbc3d442c893cedd5e7bb51d57 2023-04-27 03:56:31,712 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/a91e43513d9741c191808d3201eddfbe to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/a91e43513d9741c191808d3201eddfbe 2023-04-27 03:56:31,713 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/8d2889461cf74684a7fc54f511c18800 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/8d2889461cf74684a7fc54f511c18800 2023-04-27 03:56:31,714 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/ee4b92ebd1234f17aab6243f8f9f97d3 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/ee4b92ebd1234f17aab6243f8f9f97d3 2023-04-27 03:56:31,715 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/611a9e30647e439b903535290cb0673f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/611a9e30647e439b903535290cb0673f 2023-04-27 03:56:31,716 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/f12f4f70032d498faccdd53ce5b28618 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/f12f4f70032d498faccdd53ce5b28618 2023-04-27 03:56:31,716 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/45047dce38654811a1c834e114f79389 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/45047dce38654811a1c834e114f79389 2023-04-27 03:56:31,717 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/d7cee918e0de4fea82cdae8b6a6b1643 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/d7cee918e0de4fea82cdae8b6a6b1643 2023-04-27 03:56:31,720 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/9b056c567a6e4ef4b9d7affb202c5666, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/0666067f538c41c9b5c2b33cc5f43751, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/20478bc9c95a4834938423a9e67e19b8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/932deb8352264c9ca4e36d4c402f5e43, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/02f10fd10b534878a63b0658211e6a38, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/48a5c1d239384ba1a55278f2707911dc, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/b9cd4b36ea0b42f6b0cf7130f052d398, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/28a6235d9a9c4bb7a102de16cf70bc10, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/310071fdb6c4440b83b1fcdcc6d9155a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/45a54d3a10074cf9a0fe196986505ce6, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/a7e7f0825c6940399537fe650f9024b8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/216f8fec746d46beb3ff7e482ce64b80, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/ed65f0e4f43f461e9f4d29fc38e9b636, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/1b673f39b43345c0a0b9531a1af2944d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/fbc923851e85401dacf54132e98fc15c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/ea8543f0e7064da48b05e4ad2838c81d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/b86fb1287bb843f7ad6b6af11e275270, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/998e4efe5a1742a9998a6135dbaf037d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/31bd92da3cb5413ea1ddd1d4d36f9122, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/b07ea9d691bc45bb91a97c337c415045, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/fde2c83b615c4282aad592d98efe8932, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/7dd616c47e23491597947e082e84eb6e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/c165aef8878741f69e64bf2f98f7acb4, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/65beffef189346548550bb244840af9a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/cdae90ce064c4870b89bba23388aa807, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/7de020e9a7f34c309f294667389c7241, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/9006d5cdeb1a4aebae292e9809884531, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/c6f77fb963ca44e182608848d1180758, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/dc241d561b19441e9405a1086a449584, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/59f63a26c64b4bbfbdf339ae5f4620f8, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/a82ae29c035c458a8d8b462414921672, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/1265458bb748416bb266af93e998f1f2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/38c018d2a1fc4b68b7d276b432fabf25, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/0becb8ea67bb489fb4111fc6ee262502, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/47788a00420949bc87ce07ad86cc8908, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/4dba49bc6b01417a87c9aa8b9c2d46c3, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/bbbf65d961a04fdbaf0684a2366d99ec] to archive 2023-04-27 03:56:31,720 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 03:56:31,721 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/9b056c567a6e4ef4b9d7affb202c5666 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/9b056c567a6e4ef4b9d7affb202c5666 2023-04-27 03:56:31,722 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/0666067f538c41c9b5c2b33cc5f43751 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/0666067f538c41c9b5c2b33cc5f43751 2023-04-27 03:56:31,723 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/20478bc9c95a4834938423a9e67e19b8 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/20478bc9c95a4834938423a9e67e19b8 2023-04-27 03:56:31,724 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/932deb8352264c9ca4e36d4c402f5e43 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/932deb8352264c9ca4e36d4c402f5e43 2023-04-27 03:56:31,724 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/02f10fd10b534878a63b0658211e6a38 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/02f10fd10b534878a63b0658211e6a38 2023-04-27 03:56:31,725 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/48a5c1d239384ba1a55278f2707911dc to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/48a5c1d239384ba1a55278f2707911dc 2023-04-27 03:56:31,726 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/b9cd4b36ea0b42f6b0cf7130f052d398 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/b9cd4b36ea0b42f6b0cf7130f052d398 2023-04-27 03:56:31,726 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/28a6235d9a9c4bb7a102de16cf70bc10 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/28a6235d9a9c4bb7a102de16cf70bc10 2023-04-27 03:56:31,727 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/310071fdb6c4440b83b1fcdcc6d9155a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/310071fdb6c4440b83b1fcdcc6d9155a 2023-04-27 03:56:31,728 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/45a54d3a10074cf9a0fe196986505ce6 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/45a54d3a10074cf9a0fe196986505ce6 2023-04-27 03:56:31,728 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/a7e7f0825c6940399537fe650f9024b8 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/a7e7f0825c6940399537fe650f9024b8 2023-04-27 03:56:31,729 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/216f8fec746d46beb3ff7e482ce64b80 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/216f8fec746d46beb3ff7e482ce64b80 2023-04-27 03:56:31,730 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/ed65f0e4f43f461e9f4d29fc38e9b636 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/ed65f0e4f43f461e9f4d29fc38e9b636 2023-04-27 03:56:31,730 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/1b673f39b43345c0a0b9531a1af2944d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/1b673f39b43345c0a0b9531a1af2944d 2023-04-27 03:56:31,731 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/fbc923851e85401dacf54132e98fc15c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/fbc923851e85401dacf54132e98fc15c 2023-04-27 03:56:31,732 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/ea8543f0e7064da48b05e4ad2838c81d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/ea8543f0e7064da48b05e4ad2838c81d 2023-04-27 03:56:31,733 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/b86fb1287bb843f7ad6b6af11e275270 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/b86fb1287bb843f7ad6b6af11e275270 2023-04-27 03:56:31,733 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/998e4efe5a1742a9998a6135dbaf037d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/998e4efe5a1742a9998a6135dbaf037d 2023-04-27 03:56:31,734 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/31bd92da3cb5413ea1ddd1d4d36f9122 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/31bd92da3cb5413ea1ddd1d4d36f9122 2023-04-27 03:56:31,735 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/b07ea9d691bc45bb91a97c337c415045 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/b07ea9d691bc45bb91a97c337c415045 2023-04-27 03:56:31,736 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/fde2c83b615c4282aad592d98efe8932 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/fde2c83b615c4282aad592d98efe8932 2023-04-27 03:56:31,736 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/7dd616c47e23491597947e082e84eb6e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/7dd616c47e23491597947e082e84eb6e 2023-04-27 03:56:31,737 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/c165aef8878741f69e64bf2f98f7acb4 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/c165aef8878741f69e64bf2f98f7acb4 2023-04-27 03:56:31,738 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/65beffef189346548550bb244840af9a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/65beffef189346548550bb244840af9a 2023-04-27 03:56:31,739 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/cdae90ce064c4870b89bba23388aa807 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/cdae90ce064c4870b89bba23388aa807 2023-04-27 03:56:31,739 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/7de020e9a7f34c309f294667389c7241 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/7de020e9a7f34c309f294667389c7241 2023-04-27 03:56:31,740 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/9006d5cdeb1a4aebae292e9809884531 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/9006d5cdeb1a4aebae292e9809884531 2023-04-27 03:56:31,741 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/c6f77fb963ca44e182608848d1180758 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/c6f77fb963ca44e182608848d1180758 2023-04-27 03:56:31,742 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/dc241d561b19441e9405a1086a449584 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/dc241d561b19441e9405a1086a449584 2023-04-27 03:56:31,742 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/59f63a26c64b4bbfbdf339ae5f4620f8 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/59f63a26c64b4bbfbdf339ae5f4620f8 2023-04-27 03:56:31,743 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/a82ae29c035c458a8d8b462414921672 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/a82ae29c035c458a8d8b462414921672 2023-04-27 03:56:31,744 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/1265458bb748416bb266af93e998f1f2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/1265458bb748416bb266af93e998f1f2 2023-04-27 03:56:31,745 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/38c018d2a1fc4b68b7d276b432fabf25 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/38c018d2a1fc4b68b7d276b432fabf25 2023-04-27 03:56:31,746 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/0becb8ea67bb489fb4111fc6ee262502 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/0becb8ea67bb489fb4111fc6ee262502 2023-04-27 03:56:31,746 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/47788a00420949bc87ce07ad86cc8908 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/47788a00420949bc87ce07ad86cc8908 2023-04-27 03:56:31,747 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/4dba49bc6b01417a87c9aa8b9c2d46c3 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/4dba49bc6b01417a87c9aa8b9c2d46c3 2023-04-27 03:56:31,748 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/bbbf65d961a04fdbaf0684a2366d99ec to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/bbbf65d961a04fdbaf0684a2366d99ec 2023-04-27 03:56:31,751 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/dcbab4ee2e964367bce2defd9eb530dd, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/4897bb40512d499e90561b8c19fd87ef, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/c34472fdba6c428b8176053b2aedff7c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/c5aadee936d84e4fb683fc3a791f6a36, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/105a4e4d135f4f3eb0b350da9465f148, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/50b6bada450f4273a07b7516c6e05ce4, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/7900f9725daf442699d4a43379d5d503, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/0aad3a81e7e649fa82ad7a545a081c6d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a566b15837784089bfd58229b241268e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/8e23f2aa8d3d4153bbd3081168f72755, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a7d3b09ec7ad4e998d9939e004b7f1ef, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/b2735685c24d4c9c9a43fc815dd51354, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/13cc11afba3142b9ac6fe3d22b91c70d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/66c5c2e0bf7947a58707ba012929c349, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/04b9cb9d75a34d1c933778ed0cf1585f, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/57e0aab6b32546f4ad139a085c0f5e4e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a34a4d1591b5423d99d50884dd85855e, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/7745c74e0a96457d85ac6386ac2d370d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/9306df8ba8af40139873762d875c48d0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/d9c96c7509e34e40b7d43f6bcf30d8f0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/ad73db38881f4483aec0437c9cca4d6a, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/d4b9316ee7c64898bddab4ada5e14ef3, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/0a37f25ae60e4a2cac772f7c5de56987, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/11040a64a432490ea7a053b589445191, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/cd79675550354cf9bfb2148b5c5fb386, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/33f2ac778d57433e841795abc5e94ce2, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/042d4cd9393d424fae7a24be13462db3, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/59a1ac71baad47cf845edb6c269a2cba, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/1aa21016a1b64319b4275be3de59f242, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/dac05457821b4707a9bc14c3ad05781d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/3e85f1be105f456b8c71283e6a946871, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/3719aa86c32440cd9b2dcd461bb03eaa, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/36af05db406d492abe6f4d3c7281508c, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a6825d05a55d4892ba69427739186db0, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/17b98913bab1492f8c44f73f190e382d, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/e404a79f5d934dea96374c661f25c760] to archive 2023-04-27 03:56:31,752 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-27 03:56:31,753 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/dcbab4ee2e964367bce2defd9eb530dd to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/dcbab4ee2e964367bce2defd9eb530dd 2023-04-27 03:56:31,754 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/4897bb40512d499e90561b8c19fd87ef to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/4897bb40512d499e90561b8c19fd87ef 2023-04-27 03:56:31,755 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/c34472fdba6c428b8176053b2aedff7c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/c34472fdba6c428b8176053b2aedff7c 2023-04-27 03:56:31,756 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/c5aadee936d84e4fb683fc3a791f6a36 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/c5aadee936d84e4fb683fc3a791f6a36 2023-04-27 03:56:31,757 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/105a4e4d135f4f3eb0b350da9465f148 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/105a4e4d135f4f3eb0b350da9465f148 2023-04-27 03:56:31,757 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/50b6bada450f4273a07b7516c6e05ce4 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/50b6bada450f4273a07b7516c6e05ce4 2023-04-27 03:56:31,758 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/7900f9725daf442699d4a43379d5d503 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/7900f9725daf442699d4a43379d5d503 2023-04-27 03:56:31,759 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/0aad3a81e7e649fa82ad7a545a081c6d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/0aad3a81e7e649fa82ad7a545a081c6d 2023-04-27 03:56:31,760 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a566b15837784089bfd58229b241268e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a566b15837784089bfd58229b241268e 2023-04-27 03:56:31,761 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/8e23f2aa8d3d4153bbd3081168f72755 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/8e23f2aa8d3d4153bbd3081168f72755 2023-04-27 03:56:31,761 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a7d3b09ec7ad4e998d9939e004b7f1ef to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a7d3b09ec7ad4e998d9939e004b7f1ef 2023-04-27 03:56:31,762 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/b2735685c24d4c9c9a43fc815dd51354 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/b2735685c24d4c9c9a43fc815dd51354 2023-04-27 03:56:31,763 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/13cc11afba3142b9ac6fe3d22b91c70d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/13cc11afba3142b9ac6fe3d22b91c70d 2023-04-27 03:56:31,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-27 03:56:31,764 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/66c5c2e0bf7947a58707ba012929c349 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/66c5c2e0bf7947a58707ba012929c349 2023-04-27 03:56:31,765 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/04b9cb9d75a34d1c933778ed0cf1585f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/04b9cb9d75a34d1c933778ed0cf1585f 2023-04-27 03:56:31,766 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/57e0aab6b32546f4ad139a085c0f5e4e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/57e0aab6b32546f4ad139a085c0f5e4e 2023-04-27 03:56:31,766 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a34a4d1591b5423d99d50884dd85855e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a34a4d1591b5423d99d50884dd85855e 2023-04-27 03:56:31,767 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/7745c74e0a96457d85ac6386ac2d370d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/7745c74e0a96457d85ac6386ac2d370d 2023-04-27 03:56:31,768 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/9306df8ba8af40139873762d875c48d0 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/9306df8ba8af40139873762d875c48d0 2023-04-27 03:56:31,769 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/d9c96c7509e34e40b7d43f6bcf30d8f0 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/d9c96c7509e34e40b7d43f6bcf30d8f0 2023-04-27 03:56:31,769 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/ad73db38881f4483aec0437c9cca4d6a to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/ad73db38881f4483aec0437c9cca4d6a 2023-04-27 03:56:31,771 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/d4b9316ee7c64898bddab4ada5e14ef3 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/d4b9316ee7c64898bddab4ada5e14ef3 2023-04-27 03:56:31,772 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/0a37f25ae60e4a2cac772f7c5de56987 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/0a37f25ae60e4a2cac772f7c5de56987 2023-04-27 03:56:31,773 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/11040a64a432490ea7a053b589445191 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/11040a64a432490ea7a053b589445191 2023-04-27 03:56:31,773 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/cd79675550354cf9bfb2148b5c5fb386 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/cd79675550354cf9bfb2148b5c5fb386 2023-04-27 03:56:31,774 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/33f2ac778d57433e841795abc5e94ce2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/33f2ac778d57433e841795abc5e94ce2 2023-04-27 03:56:31,775 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/042d4cd9393d424fae7a24be13462db3 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/042d4cd9393d424fae7a24be13462db3 2023-04-27 03:56:31,776 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/59a1ac71baad47cf845edb6c269a2cba to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/59a1ac71baad47cf845edb6c269a2cba 2023-04-27 03:56:31,777 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/1aa21016a1b64319b4275be3de59f242 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/1aa21016a1b64319b4275be3de59f242 2023-04-27 03:56:31,777 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/dac05457821b4707a9bc14c3ad05781d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/dac05457821b4707a9bc14c3ad05781d 2023-04-27 03:56:31,778 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/3e85f1be105f456b8c71283e6a946871 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/3e85f1be105f456b8c71283e6a946871 2023-04-27 03:56:31,779 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/3719aa86c32440cd9b2dcd461bb03eaa to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/3719aa86c32440cd9b2dcd461bb03eaa 2023-04-27 03:56:31,780 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/36af05db406d492abe6f4d3c7281508c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/36af05db406d492abe6f4d3c7281508c 2023-04-27 03:56:31,780 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a6825d05a55d4892ba69427739186db0 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a6825d05a55d4892ba69427739186db0 2023-04-27 03:56:31,781 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/17b98913bab1492f8c44f73f190e382d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/17b98913bab1492f8c44f73f190e382d 2023-04-27 03:56:31,782 DEBUG [StoreCloser-TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/e404a79f5d934dea96374c661f25c760 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/e404a79f5d934dea96374c661f25c760 2023-04-27 03:56:31,785 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/recovered.edits/922.seqid, newMaxSeqId=922, maxSeqId=4 2023-04-27 03:56:31,786 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231. 2023-04-27 03:56:31,786 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1c25ca7c0495563171ae7ccdf0350231: 2023-04-27 03:56:31,787 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:31,788 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=1c25ca7c0495563171ae7ccdf0350231, regionState=CLOSED 2023-04-27 03:56:31,788 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682567791788"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682567791788"}]},"ts":"1682567791788"} 2023-04-27 03:56:31,789 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-27 03:56:31,790 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 1c25ca7c0495563171ae7ccdf0350231, server=jenkins-hbase4.apache.org,40885,1682567618305 in 1.1240 sec 2023-04-27 03:56:31,790 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-27 03:56:31,790 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1c25ca7c0495563171ae7ccdf0350231, UNASSIGN in 1.1260 sec 2023-04-27 03:56:31,791 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682567791791"}]},"ts":"1682567791791"} 2023-04-27 03:56:31,792 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-27 03:56:31,794 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-27 03:56:31,795 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.1370 sec 2023-04-27 03:56:32,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-27 03:56:32,764 INFO [Listener at localhost/33933] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-27 03:56:32,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-27 03:56:32,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:56:32,767 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:56:32,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-27 03:56:32,768 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:56:32,772 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:32,774 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A, FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B, FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C, FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/recovered.edits] 2023-04-27 03:56:32,776 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/82aa26af80964c53b27c531e7110e0f5 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/82aa26af80964c53b27c531e7110e0f5 2023-04-27 03:56:32,777 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/859f74522f9a4ac0b409c0833a77b861 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/859f74522f9a4ac0b409c0833a77b861 2023-04-27 03:56:32,779 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/b04e2e99fab742f885a72a1c23fda382 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/A/b04e2e99fab742f885a72a1c23fda382 2023-04-27 03:56:32,781 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/05983e22d37f41889ccf7e3946e9c1ee to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/05983e22d37f41889ccf7e3946e9c1ee 2023-04-27 03:56:32,782 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/6fa5aa8ff3fe4ca2a625b958319df65b to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/6fa5aa8ff3fe4ca2a625b958319df65b 2023-04-27 03:56:32,783 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/d79c39e970be44c4b0b280f1ca32954b to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/B/d79c39e970be44c4b0b280f1ca32954b 2023-04-27 03:56:32,784 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/0fa41d44e3514fd78b7de0dac19da3ca to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/0fa41d44e3514fd78b7de0dac19da3ca 2023-04-27 03:56:32,785 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a107ce5e72264d288567b7c21ecb1146 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/a107ce5e72264d288567b7c21ecb1146 2023-04-27 03:56:32,786 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/ccd307361cf34cf58a3cda06abe22538 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/C/ccd307361cf34cf58a3cda06abe22538 2023-04-27 03:56:32,789 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/recovered.edits/922.seqid to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231/recovered.edits/922.seqid 2023-04-27 03:56:32,789 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/.tmp/data/default/TestAcidGuarantees/1c25ca7c0495563171ae7ccdf0350231 2023-04-27 03:56:32,789 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-27 03:56:32,789 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-27 03:56:32,790 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-27 03:56:32,792 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427094541c0d4694af4831a491c746cbfb0 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427094541c0d4694af4831a491c746cbfb0 2023-04-27 03:56:32,793 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304270f0087ef994a4ba8a69bd4135ebec207 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304270f0087ef994a4ba8a69bd4135ebec207 2023-04-27 03:56:32,794 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304270f21f73eddae4d6cbae67e3a1c103780 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304270f21f73eddae4d6cbae67e3a1c103780 2023-04-27 03:56:32,795 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042710237b1c662d423c95691d7496b51531 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042710237b1c662d423c95691d7496b51531 2023-04-27 03:56:32,796 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042726c9cb05cb29402ebb8c344508b00669 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042726c9cb05cb29402ebb8c344508b00669 2023-04-27 03:56:32,797 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272d7256610b8447dc8e4b6765aa58203f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304272d7256610b8447dc8e4b6765aa58203f 2023-04-27 03:56:32,798 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427308f356e5d094e7f93fd66ad66561f74 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427308f356e5d094e7f93fd66ad66561f74 2023-04-27 03:56:32,798 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042733db045108874f65a19eda56ca9659c6 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042733db045108874f65a19eda56ca9659c6 2023-04-27 03:56:32,799 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427340bb7073b5c49638ca9c504c50aea4d to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427340bb7073b5c49638ca9c504c50aea4d 2023-04-27 03:56:32,800 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042745e691e0e55d4b56b855aada087661ac to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042745e691e0e55d4b56b855aada087661ac 2023-04-27 03:56:32,801 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042751dbcfa392424adb9bcaa9a59f8fe9d6 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042751dbcfa392424adb9bcaa9a59f8fe9d6 2023-04-27 03:56:32,802 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304275fdc8a4a8fb54b0e9e870f4b1e63b0f3 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304275fdc8a4a8fb54b0e9e870f4b1e63b0f3 2023-04-27 03:56:32,803 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304276c335d1460c54ba3b7305bc0d612a97f to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304276c335d1460c54ba3b7305bc0d612a97f 2023-04-27 03:56:32,804 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042778050e411c14453aa30f3a5cc36cbe33 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042778050e411c14453aa30f3a5cc36cbe33 2023-04-27 03:56:32,805 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304277a24a946976146ee9259062680d87aee to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304277a24a946976146ee9259062680d87aee 2023-04-27 03:56:32,806 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304278dfe2c0b8d544ca7aea24a3e7931deae to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304278dfe2c0b8d544ca7aea24a3e7931deae 2023-04-27 03:56:32,807 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427a1333c6a3247463b830ad044b0447cfd to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427a1333c6a3247463b830ad044b0447cfd 2023-04-27 03:56:32,808 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427b645897874e64074a37bc94f7170ffa2 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427b645897874e64074a37bc94f7170ffa2 2023-04-27 03:56:32,809 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427bc911190f2824a3ca5c2b6c753a2be87 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427bc911190f2824a3ca5c2b6c753a2be87 2023-04-27 03:56:32,809 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427be24d603aedc4001ae38e9b95f2cf820 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427be24d603aedc4001ae38e9b95f2cf820 2023-04-27 03:56:32,810 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427bf91b69429af4993ad424a923886e925 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427bf91b69429af4993ad424a923886e925 2023-04-27 03:56:32,811 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427c18d02f0503a49e6ac3c4ca39010afd5 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427c18d02f0503a49e6ac3c4ca39010afd5 2023-04-27 03:56:32,812 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427cf41e38b1ad749bd948a892d1e38096e to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427cf41e38b1ad749bd948a892d1e38096e 2023-04-27 03:56:32,813 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427d33efab45de248c9819a89c2a43a3722 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427d33efab45de248c9819a89c2a43a3722 2023-04-27 03:56:32,813 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427da0893d87860438ebaa4c717a522616c to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427da0893d87860438ebaa4c717a522616c 2023-04-27 03:56:32,814 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427de470a56c3344e9f9178147dd3bcfa79 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427de470a56c3344e9f9178147dd3bcfa79 2023-04-27 03:56:32,815 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427e69f72756a8147e99072dba6f8dc61ff to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427e69f72756a8147e99072dba6f8dc61ff 2023-04-27 03:56:32,816 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427ea9878a2842641e6ba33b69354170b46 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427ea9878a2842641e6ba33b69354170b46 2023-04-27 03:56:32,816 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427eae0c32fc08b4f66b85c3b022e8fbe62 to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427eae0c32fc08b4f66b85c3b022e8fbe62 2023-04-27 03:56:32,817 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427f8c8cc52dd0b410781114494724a0fcd to hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230427f8c8cc52dd0b410781114494724a0fcd 2023-04-27 03:56:32,818 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-27 03:56:32,819 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:56:32,820 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-27 03:56:32,822 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-27 03:56:32,822 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:56:32,822 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-27 03:56:32,823 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682567792822"}]},"ts":"9223372036854775807"} 2023-04-27 03:56:32,824 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-27 03:56:32,824 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1c25ca7c0495563171ae7ccdf0350231, NAME => 'TestAcidGuarantees,,1682567759215.1c25ca7c0495563171ae7ccdf0350231.', STARTKEY => '', ENDKEY => ''}] 2023-04-27 03:56:32,824 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-27 03:56:32,824 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682567792824"}]},"ts":"9223372036854775807"} 2023-04-27 03:56:32,825 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-27 03:56:32,827 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-27 03:56:32,827 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 62 msec 2023-04-27 03:56:32,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39131] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-27 03:56:32,869 INFO [Listener at localhost/33933] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-27 03:56:32,883 INFO [Listener at localhost/33933] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 228), OpenFileDescriptor=571 (was 566) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=376 (was 371) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=12315 (was 12288) - AvailableMemoryMB LEAK? - 2023-04-27 03:56:32,883 INFO [Listener at localhost/33933] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-27 03:56:32,883 INFO [Listener at localhost/33933] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-27 03:56:32,884 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3ddd91d3 to 127.0.0.1:55020 2023-04-27 03:56:32,884 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:56:32,884 DEBUG [Listener at localhost/33933] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-27 03:56:32,884 DEBUG [Listener at localhost/33933] util.JVMClusterUtil(257): Found active master hash=969248687, stopped=false 2023-04-27 03:56:32,884 INFO [Listener at localhost/33933] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,39131,1682567617081 2023-04-27 03:56:32,885 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-27 03:56:32,885 INFO [Listener at localhost/33933] procedure2.ProcedureExecutor(629): Stopping 2023-04-27 03:56:32,885 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-27 03:56:32,885 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-27 03:56:32,886 DEBUG [Listener at localhost/33933] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x53fd259f to 127.0.0.1:55020 2023-04-27 03:56:32,886 DEBUG [Listener at localhost/33933] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:56:32,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-27 03:56:32,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-27 03:56:32,886 INFO [Listener at localhost/33933] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,40885,1682567618305' ***** 2023-04-27 03:56:32,886 INFO [Listener at localhost/33933] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-27 03:56:32,886 INFO [RS:0;jenkins-hbase4:40885] regionserver.HeapMemoryManager(220): Stopping 2023-04-27 03:56:32,887 INFO [RS:0;jenkins-hbase4:40885] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-27 03:56:32,887 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-27 03:56:32,887 INFO [RS:0;jenkins-hbase4:40885] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-27 03:56:32,887 INFO [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(3304): Received CLOSE for bc58fe2fa2408f679ad2d4ab14224359 2023-04-27 03:56:32,888 INFO [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:32,888 DEBUG [RS:0;jenkins-hbase4:40885] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x006e5da7 to 127.0.0.1:55020 2023-04-27 03:56:32,888 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing bc58fe2fa2408f679ad2d4ab14224359, disabling compactions & flushes 2023-04-27 03:56:32,888 DEBUG [RS:0;jenkins-hbase4:40885] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:56:32,888 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682567620670.bc58fe2fa2408f679ad2d4ab14224359. 2023-04-27 03:56:32,889 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682567620670.bc58fe2fa2408f679ad2d4ab14224359. 2023-04-27 03:56:32,889 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682567620670.bc58fe2fa2408f679ad2d4ab14224359. after waiting 0 ms 2023-04-27 03:56:32,889 INFO [RS:0;jenkins-hbase4:40885] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-27 03:56:32,889 INFO [RS:0;jenkins-hbase4:40885] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-27 03:56:32,889 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682567620670.bc58fe2fa2408f679ad2d4ab14224359. 2023-04-27 03:56:32,889 INFO [RS:0;jenkins-hbase4:40885] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-27 03:56:32,889 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing bc58fe2fa2408f679ad2d4ab14224359 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-27 03:56:32,889 INFO [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-27 03:56:32,889 INFO [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-27 03:56:32,889 DEBUG [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, bc58fe2fa2408f679ad2d4ab14224359=hbase:namespace,,1682567620670.bc58fe2fa2408f679ad2d4ab14224359.} 2023-04-27 03:56:32,890 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-27 03:56:32,890 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-27 03:56:32,890 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-27 03:56:32,890 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-27 03:56:32,890 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-27 03:56:32,890 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-27 03:56:32,891 DEBUG [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(1505): Waiting on 1588230740, bc58fe2fa2408f679ad2d4ab14224359 2023-04-27 03:56:32,942 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-27 03:56:33,091 DEBUG [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(1505): Waiting on 1588230740, bc58fe2fa2408f679ad2d4ab14224359 2023-04-27 03:56:33,292 DEBUG [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(1505): Waiting on 1588230740, bc58fe2fa2408f679ad2d4ab14224359 2023-04-27 03:56:33,302 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/namespace/bc58fe2fa2408f679ad2d4ab14224359/.tmp/info/46e34e4b6ebf4892a9bfb46de8146bab 2023-04-27 03:56:33,302 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740/.tmp/info/cfb94336be704018a0da2ad76bdb36a4 2023-04-27 03:56:33,306 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/namespace/bc58fe2fa2408f679ad2d4ab14224359/.tmp/info/46e34e4b6ebf4892a9bfb46de8146bab as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/namespace/bc58fe2fa2408f679ad2d4ab14224359/info/46e34e4b6ebf4892a9bfb46de8146bab 2023-04-27 03:56:33,306 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for cfb94336be704018a0da2ad76bdb36a4 2023-04-27 03:56:33,310 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/namespace/bc58fe2fa2408f679ad2d4ab14224359/info/46e34e4b6ebf4892a9bfb46de8146bab, entries=2, sequenceid=6, filesize=4.8 K 2023-04-27 03:56:33,311 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for bc58fe2fa2408f679ad2d4ab14224359 in 422ms, sequenceid=6, compaction requested=false 2023-04-27 03:56:33,315 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/namespace/bc58fe2fa2408f679ad2d4ab14224359/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-27 03:56:33,316 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682567620670.bc58fe2fa2408f679ad2d4ab14224359. 2023-04-27 03:56:33,316 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for bc58fe2fa2408f679ad2d4ab14224359: 2023-04-27 03:56:33,316 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1682567620670.bc58fe2fa2408f679ad2d4ab14224359. 2023-04-27 03:56:33,492 DEBUG [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-27 03:56:33,692 DEBUG [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-27 03:56:33,713 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740/.tmp/rep_barrier/f90d4aaf311641178606ce457b8ed50e 2023-04-27 03:56:33,716 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f90d4aaf311641178606ce457b8ed50e 2023-04-27 03:56:33,732 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-27 03:56:33,733 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-27 03:56:33,892 INFO [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-04-27 03:56:33,893 DEBUG [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-04-27 03:56:33,893 DEBUG [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-27 03:56:34,093 DEBUG [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-27 03:56:34,122 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740/.tmp/table/95b7acb8d0234932a03704a98a6756b4 2023-04-27 03:56:34,124 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 95b7acb8d0234932a03704a98a6756b4 2023-04-27 03:56:34,125 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740/.tmp/info/cfb94336be704018a0da2ad76bdb36a4 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740/info/cfb94336be704018a0da2ad76bdb36a4 2023-04-27 03:56:34,128 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for cfb94336be704018a0da2ad76bdb36a4 2023-04-27 03:56:34,128 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740/info/cfb94336be704018a0da2ad76bdb36a4, entries=22, sequenceid=93, filesize=7.1 K 2023-04-27 03:56:34,129 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740/.tmp/rep_barrier/f90d4aaf311641178606ce457b8ed50e as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740/rep_barrier/f90d4aaf311641178606ce457b8ed50e 2023-04-27 03:56:34,132 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f90d4aaf311641178606ce457b8ed50e 2023-04-27 03:56:34,132 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740/rep_barrier/f90d4aaf311641178606ce457b8ed50e, entries=6, sequenceid=93, filesize=5.5 K 2023-04-27 03:56:34,132 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740/.tmp/table/95b7acb8d0234932a03704a98a6756b4 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740/table/95b7acb8d0234932a03704a98a6756b4 2023-04-27 03:56:34,135 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 95b7acb8d0234932a03704a98a6756b4 2023-04-27 03:56:34,135 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740/table/95b7acb8d0234932a03704a98a6756b4, entries=9, sequenceid=93, filesize=5.5 K 2023-04-27 03:56:34,136 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 1246ms, sequenceid=93, compaction requested=false 2023-04-27 03:56:34,143 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-27 03:56:34,143 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-27 03:56:34,144 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-27 03:56:34,144 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-27 03:56:34,144 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-27 03:56:34,293 INFO [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,40885,1682567618305; all regions closed. 2023-04-27 03:56:34,301 DEBUG [RS:0;jenkins-hbase4:40885] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/oldWALs 2023-04-27 03:56:34,301 INFO [RS:0;jenkins-hbase4:40885] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C40885%2C1682567618305.meta:.meta(num 1682567620392) 2023-04-27 03:56:34,305 DEBUG [RS:0;jenkins-hbase4:40885] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/oldWALs 2023-04-27 03:56:34,305 INFO [RS:0;jenkins-hbase4:40885] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C40885%2C1682567618305:(num 1682567619893) 2023-04-27 03:56:34,305 DEBUG [RS:0;jenkins-hbase4:40885] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:56:34,305 INFO [RS:0;jenkins-hbase4:40885] regionserver.LeaseManager(133): Closed leases 2023-04-27 03:56:34,306 INFO [RS:0;jenkins-hbase4:40885] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-27 03:56:34,306 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-27 03:56:34,308 INFO [RS:0;jenkins-hbase4:40885] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:40885 2023-04-27 03:56:34,313 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,40885,1682567618305 2023-04-27 03:56:34,313 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-27 03:56:34,313 ERROR [Listener at localhost/33933-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@62232017 rejected from java.util.concurrent.ThreadPoolExecutor@661cb88b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3082] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-27 03:56:34,314 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-27 03:56:34,314 ERROR [Listener at localhost/33933-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@692f4ebf rejected from java.util.concurrent.ThreadPoolExecutor@661cb88b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3082] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-27 03:56:34,314 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,40885,1682567618305] 2023-04-27 03:56:34,315 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,40885,1682567618305; numProcessing=1 2023-04-27 03:56:34,316 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,40885,1682567618305 already deleted, retry=false 2023-04-27 03:56:34,316 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,40885,1682567618305 expired; onlineServers=0 2023-04-27 03:56:34,316 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,39131,1682567617081' ***** 2023-04-27 03:56:34,317 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-27 03:56:34,317 DEBUG [M:0;jenkins-hbase4:39131] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78f88594, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-27 03:56:34,317 INFO [M:0;jenkins-hbase4:39131] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,39131,1682567617081 2023-04-27 03:56:34,317 INFO [M:0;jenkins-hbase4:39131] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,39131,1682567617081; all regions closed. 2023-04-27 03:56:34,317 DEBUG [M:0;jenkins-hbase4:39131] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-27 03:56:34,317 DEBUG [M:0;jenkins-hbase4:39131] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-27 03:56:34,317 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-27 03:56:34,317 DEBUG [M:0;jenkins-hbase4:39131] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-27 03:56:34,317 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682567619516] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682567619516,5,FailOnTimeoutGroup] 2023-04-27 03:56:34,317 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682567619515] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682567619515,5,FailOnTimeoutGroup] 2023-04-27 03:56:34,317 INFO [M:0;jenkins-hbase4:39131] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-27 03:56:34,317 INFO [M:0;jenkins-hbase4:39131] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-27 03:56:34,317 INFO [M:0;jenkins-hbase4:39131] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-27 03:56:34,318 DEBUG [M:0;jenkins-hbase4:39131] master.HMaster(1512): Stopping service threads 2023-04-27 03:56:34,318 INFO [M:0;jenkins-hbase4:39131] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-27 03:56:34,318 ERROR [M:0;jenkins-hbase4:39131] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[IPC Client (1317398507) connection to localhost/127.0.0.1:33677 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-27 03:56:34,319 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-27 03:56:34,319 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-27 03:56:34,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-27 03:56:34,319 INFO [M:0;jenkins-hbase4:39131] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-27 03:56:34,319 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-27 03:56:34,320 DEBUG [M:0;jenkins-hbase4:39131] zookeeper.ZKUtil(398): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-27 03:56:34,320 WARN [M:0;jenkins-hbase4:39131] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-27 03:56:34,320 INFO [M:0;jenkins-hbase4:39131] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-27 03:56:34,320 INFO [M:0;jenkins-hbase4:39131] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-27 03:56:34,321 DEBUG [M:0;jenkins-hbase4:39131] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-27 03:56:34,321 INFO [M:0;jenkins-hbase4:39131] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-27 03:56:34,321 DEBUG [M:0;jenkins-hbase4:39131] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-27 03:56:34,321 DEBUG [M:0;jenkins-hbase4:39131] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-27 03:56:34,321 DEBUG [M:0;jenkins-hbase4:39131] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-27 03:56:34,321 INFO [M:0;jenkins-hbase4:39131] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.84 KB heapSize=393.62 KB 2023-04-27 03:56:34,415 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-27 03:56:34,415 INFO [RS:0;jenkins-hbase4:40885] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,40885,1682567618305; zookeeper connection closed. 2023-04-27 03:56:34,415 ERROR [Listener at localhost/33933-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@112cd220 rejected from java.util.concurrent.ThreadPoolExecutor@661cb88b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3082] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-27 03:56:34,416 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): regionserver:40885-0x101c3c040af0001, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-27 03:56:34,416 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7a5cd6b9] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7a5cd6b9 2023-04-27 03:56:34,416 ERROR [Listener at localhost/33933-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@50f8310c rejected from java.util.concurrent.ThreadPoolExecutor@661cb88b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3082] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-27 03:56:34,417 INFO [Listener at localhost/33933] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-27 03:56:34,735 INFO [M:0;jenkins-hbase4:39131] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.84 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/e20eaba00aab47efa268496827069c46 2023-04-27 03:56:34,738 INFO [M:0;jenkins-hbase4:39131] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e20eaba00aab47efa268496827069c46 2023-04-27 03:56:34,739 DEBUG [M:0;jenkins-hbase4:39131] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/e20eaba00aab47efa268496827069c46 as hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/e20eaba00aab47efa268496827069c46 2023-04-27 03:56:34,742 INFO [M:0;jenkins-hbase4:39131] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e20eaba00aab47efa268496827069c46 2023-04-27 03:56:34,742 INFO [M:0;jenkins-hbase4:39131] regionserver.HStore(1080): Added hdfs://localhost:33677/user/jenkins/test-data/2f598bc4-12c0-0d8e-4a40-a387cdda2421/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/e20eaba00aab47efa268496827069c46, entries=65, sequenceid=578, filesize=24.7 K 2023-04-27 03:56:34,742 INFO [M:0;jenkins-hbase4:39131] regionserver.HRegion(2948): Finished flush of dataSize ~344.84 KB/353121, heapSize ~393.60 KB/403048, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 421ms, sequenceid=578, compaction requested=false 2023-04-27 03:56:34,744 INFO [M:0;jenkins-hbase4:39131] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-27 03:56:34,744 DEBUG [M:0;jenkins-hbase4:39131] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-27 03:56:34,747 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-27 03:56:34,747 INFO [M:0;jenkins-hbase4:39131] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-27 03:56:34,747 INFO [M:0;jenkins-hbase4:39131] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39131 2023-04-27 03:56:34,749 DEBUG [M:0;jenkins-hbase4:39131] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,39131,1682567617081 already deleted, retry=false 2023-04-27 03:56:34,850 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-27 03:56:34,850 INFO [M:0;jenkins-hbase4:39131] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,39131,1682567617081; zookeeper connection closed. 2023-04-27 03:56:34,850 ERROR [Listener at localhost/33933-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@58a08889 rejected from java.util.concurrent.ThreadPoolExecutor@7876535c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5150] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-27 03:56:34,851 DEBUG [Listener at localhost/33933-EventThread] zookeeper.ZKWatcher(600): master:39131-0x101c3c040af0000, quorum=127.0.0.1:55020, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-27 03:56:34,851 ERROR [Listener at localhost/33933-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@75ec9fd8 rejected from java.util.concurrent.ThreadPoolExecutor@7876535c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5150] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-27 03:56:34,852 WARN [Listener at localhost/33933] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-27 03:56:34,856 INFO [Listener at localhost/33933] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-27 03:56:34,963 WARN [BP-756948239-172.31.14.131-1682567613764 heartbeating to localhost/127.0.0.1:33677] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-27 03:56:34,963 WARN [BP-756948239-172.31.14.131-1682567613764 heartbeating to localhost/127.0.0.1:33677] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-756948239-172.31.14.131-1682567613764 (Datanode Uuid 689edc05-2e05-4ce0-af8a-6f976cd8f04d) service to localhost/127.0.0.1:33677 2023-04-27 03:56:34,966 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/cluster_10dc1395-631d-3033-a522-9ded50d65883/dfs/data/data1/current/BP-756948239-172.31.14.131-1682567613764] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-27 03:56:34,966 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2be4cf49-d13c-4079-fca6-567600c86b19/cluster_10dc1395-631d-3033-a522-9ded50d65883/dfs/data/data2/current/BP-756948239-172.31.14.131-1682567613764] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-27 03:56:34,998 INFO [Listener at localhost/33933] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-27 03:56:35,178 INFO [Listener at localhost/33933] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-27 03:56:35,363 INFO [Listener at localhost/33933] hbase.HBaseTestingUtility(1293): Minicluster is down